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Abstract 

Over  the  years,  the  reliability  of  fielded  weapon 
systems  has  consistently  been  less  than  what  was  predicted. 

In  the  area  of  avionics  equipment,  the  reliability  gap 
between  "predicted  mean-time-between-failures  (MTBFs)"  and 
"field  MTBFs"  was  reported  to  range  from  7:1  to  20:1 
(39:231).  The  inability  to  accurately  relate  factory 
(whether  specified,  predicted,  or  demonstrated)  reliability 
to  the  field  reliability  of  weapon  systems/subsystems  has  a 
significant  impact  on  operational  readiness  and  life  cycle 
costs. 

To  study  the  reliability  gap  between  factory  and  field 
reliability,  this  research  is  divided  into  three  distinct 
phases',  with  the  following  objectives: 

1.  To  examine  the  existence  of  the  reliability  gap  in 
airborne  tactical  radars. 

2.  To  identify  and  analyze  the  major  contributors  to 
the  reliability  gap.  Specifically,  to  identify  the 
most  significant  contributors. 

3.  To  explore  practical  guidelines  on  how  to  minimize 
the  reliability  gap.  Specifically,  to  identify  some 
practical  guidelines  on  reliability  growth. 

To  examine  the  existence  of  the  reliability  gap,  this 
thesis  obtained  the  data  from  the  Naval  Air  Systems  Command 
(NAVAIR)  and  examined  the  radars  on  Air  Force  and  Navy 
aircraft.  Using  experts '/Practitioners  *  (DoD  <S  Industry) 


x 


opinions,  contributors  to  the  reliability  gap  were 
identified.  In  addition,  some  practical  guidelines  on 
reliability  growth  management  were  also  identified.  This 
thesis  consolidated  some  of  the  best  currently  available 
thinking  cn  the  major  contributors  to  the  reliability  gap 
and  some  of  the  practical  guidelines  on  reliability  growth. 
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EXPERTS'  OPINIONS  ON  THE  PELIABILITY  GAP 


AND  SOME  PRACTICAL  GUIDELINES  ON 
RELIABILITY  GROWTH 

I .  Introduction 

Over  the  years,  the  reliability  of  fielded  weapon 

systems  has  consistently  been  less  than  what  was  predicted 

or  demonstrated.  In  the  area  of  avionics  equipment: 

...  it  has  been  reported  that  the  ratio  of  an 
equipment's  demonstrated  MTBF  (mean  time  between 
failure)  to  its  field  MTBF  can  range  from  7:1  to 
20:1.  Even  greater  disparities  are  quoted  when 
comparing  predicted  MTBFs  to  field  MTBFs . 

[38:231] 

The  field  reliability  has  generally  failed  to  measure  up 
to  the  factory  reliability  (whether  specified,  predicted,  or 
demonstrated)  (44:177).  The  inability  to  relate  field 
reliability  to  factory  reliability  has  shown  to  have  a 
significant  impact  on  system  operational  readiness 
and  life  cycle  costs  (16,  40,  44,  48,  58). 

This  chapter  introduces  the  research  issues  concerning 
the  existence,  the  major  contributors,  and  the  minimizing 
approaches  to  the  reliability  gap  between  field  and  factory 
reliability.  The  chapter  begins  with  a.  discussion  on  the 
background  of  the  research  topic  followed  by  a  description 
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of  the  specific  problem.  The  scope  and  limitations  of  this 
research  are  then  presented  along  with  the  research 
objectives,  research  questions,  and  potential  benefits.  The 
chapter  concludes  with  an  overview  of  the  remaining  chapters 
in  this  thesis. 


Background 

During  the  past  few  years,  numerous  studies  have 

indicated  a  wide  discrepancy  between  the  field  MTBFs , 

measured  under  actual  operational  environment,  and  the 

predicted  MTBFs.  In  one  of  the  studies: 

The  U.S.  Comptroller  General  states  that  'one  of 
the  persistent  problems  with  weapons  systems  has 
been  the  discrepancy  between  contractually 
specified  reliability  goals  and  those  encountered 
in  the  operational  environment'.  U.S.  Comptroller 
recommends  that  'The' services  continue  to  strive 
to  narrow  the  gap  between  contractually  specified 
reliability  and  maintainability  and  those  factors 
measurable  under  operational  conditions ' .  In 
reviewing  a  current  program,  it  notes  that  'The 
test  conditions  are  not  fully  representative  of  the 
operational  environment.  Only  RELEVANT  failures 
are  counted.  For  example,  one-time  intermittent 
failures  whose  cause  could  not  be  determined  were 
not  counted.  Failures  caused  by  accidental  damage, 
operator  error,  etc.,  were  not  considered.  In 
the  field,  however,  these  items  create  significant 
workloads  and  spares  demands,  particularly  in  the 
avionics  areas.'  [54:5] 

In  the  area  of  airborne  avionics,  Kern  (38)  studied  the 
relationships  between  field  and  factory  MTBFs  (whether 
specified,  predicted,  or  demonstrated  values)  of  16 
different  pieces  of  avionics  equipment  from  10  different 
aircraft  and  found  the  ratio  of  the  disparity  to  range 
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from  2.1:1  to  9.1:1  (38:231).  Furthermore,  Montemayor  (54) 
studied  the  reliability  gap  between  predicted  and  field 
reliability  of  airborne  radars  and  found  the  ratio  of 
predicted  reliability  to  field  reliability  to  be  5:1 
(44:177).  Table  1  summarizes  some  of  the  historical 
findings  on  the  disparity  between  predicted  or  demonstrated 
reliability  and  field  reliability. 


Table  1.  Historical  Findings  on  the  Difference 
Between  the  Predicted  or  Demonstrated 
Reliability  and  Field  Reliability  (44:177) 


Equipment  Reliability  Ratio  Reliability  Ratio 

Predicted  :  Field  Demonstrated  :  Field 


Airborne 

Avionics 

>20:1 

Airborne 

Transport 

2.3:1 

2.1:1 

Airborne 

Composite 

7.7:1 

5.9:1 

Airborne 

Fighter 

9.1:1 

6.7:1 

Airborne 

Radars 

5.0:1 

Specific  Research  Problem 

The  difference  (gap)  between  the  field  and  the  factory 
reliability  (whether  specified,  predicted,  or  demonstrated) 
has  been  documented  in  numerous  studies.  The  ability  to 
accurately  predict  the  reliability  of  fielded  weapon  systems 
has  shown  to  be  useful  in  many  areas.  Among  them  is  the 
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ability  to  make  "economic  provision  for  repair  costs,  spare 
parts  inventories,  system  availability,  operational 
effectiveness...."  (33:108).  Within  the  military,  there  is 
a  need  to  relate  the  factory  reliability  (whether  predicted, 
specified,  or  demonstrated)  to  the  actual  field  reliability 
in  order  to: 

...establish  realistic  quantitative  reliability 
and  maintainability  requirements  and  estimate 
weapon  system  readiness  and  maintenance  resource 
demands.  [44:177] 

The  inability  to  accurately  predict  the  reliability  of 
fielded  weapon  systems  has  been  proven  to  be  costly. 

Kolston  (40)  found  that.: 

...initial  spares  provisioning  and  procurement  of 
support  equipment  based  on  inaccurate  predictions 
of  reliability  characteristics  may  result  in 
non-optimum  allocation  of  program  resources,  and, 
in  turn,  low  levels  of  operational  readiness. 
[40:245] 

In  examining  the  field  reliability  of  deployed  radars, 

Cougan  and  Kindig  (16)  found  that  without  the  proper 
reliability  emphasis,  more  than  800  deployed  radars 
"exhibited  unacceptably  low  field  reliability  and 
operational  readiness,  accompanied  by  high  life  cycle  costs" 
(16:121).  Due  to  the  increasing  logistic  support  costs, 
there  is  an  increased  interest  in  identifying  the  causal 
factors  to  the  differences,  "as  well  as  a  requirement  for 
better  predictive  methods  for  estimating  field  operational 
reliability"  (38:231). 
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In  the  interest  of  maximizing  the  operational  readiness 
and  optimizing  the  life  cycle  costs,  this  research 
investigates  the  reliability  gap.  Figure  1  presents  the 
generic  reliability  gap  between  the  factory  and  field 
reliability.  The  existence  of  the  gap  is  well  documented 
in  numerous  studies.  Research  is  needed  to  identify  the 
causal  factors  to  the  gap,  and  to  explore!  ways  of  minimizing 
the  gap,  and  consequently  increase  operational  readiness  and 
optimize  life  cycle  costs. 
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Figure  1.  The  Reliability  Gap 
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The  extent  of  this  research  will  be  divided  into  three 
distinct  phases.  In  the  first  phase,  tactical  airborne 
radars  will  be  examined  to  present  the  existence  of  the 
reliability  gap.  The  courtesy  slide  from  NAVAIR  consists  of 
Air  Force  as  well  as  Navy  aircraft.  It  is  the  researcher's 
intent  that  this  part  of  the  study  be  used  primarily  to 
better  understand  the  reliability  problems.  Consequently, 
specific  systems  will  not  be  named.  In  the  second  phase, 
inter-views  will  be  used  to  collect  experts  '/practitioners ' 
opinions  to  identify  and  rank  order  the  major  contributors 
to  the  reliability  gap.  Reliability  experts/practitioners 
from  the  Army,  Navy,  Air  Force,  and  Industry  will  be  asked 
to  participate  in  this  part  of  the  study.  Personal,  as  well 
as  telephone,  interviews  will  be  conducted.  In  addition,  a 
second  round  of  interviews  will  be  conducted  to  validate  the 
findings  from  the  first  round.  Finally,  in  the  third  phase, 
practical  guidelines  (experts '/practitioners' )  will  be 
examined  on  how  to  minimize  the  gap.  Specifically,  some 
practical  guidelines  on  reliability  prediction  and 
reliability  growth  will  be  examined.  The  data  source  for 
this  phase  will  be  the  experts '/practitioners '  opinions, 
various  studies  conducted  by  the  experts/practitioners,  and 
appropriate  historical  studies. 
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The  limitations  associated  with  the  scope  of  this 
research  are  as  follows: 

a.  The  selection  of  the  participants  in  each  category 
(Army,  Navy,  Air  Force,  and  Industry)  of  the  interviews 
will  be  based  on  the  availability  of  the  experts/ 
practitioners  and  will  not  necessarily  be  proportionately 
representative  of  the  population  at  large.  Consequently,  a 
similar  study  using  another  sample  from  the  target 
population  could  produce  different  results. 

b.  xhe  data  obtained  in  this  study  will  be  derived  from 
personal  and  telephone  interviews  of  experts.  As  a  result, 
data  obtained  in  this  manner  are  subjective  in  nature  and 
could  not  be  quantitatively  verified.  Whenever  interviews 
are  involved,  there  exists  the  possibility  of  interjecting 
bias.  The  researchers  can  interject  bias  into  the  process  by 
their  presence,  by  the  way  questions  are  presented,  or  by  a 
host  of  other  factors  (26:165-167).  Although  success  could 
not  be  verified,  every  effort  will  be  made  to  eliminate  or 
reduce  bias . 

c.  Alternative  techniques  will  be  selected  to  minimize 
the  reliability  gap  and  the  techniques  selected  are  by  no 
means  all-inclusive.  The  intent  for  this  phase  of  the  study 
is  to  draw  on  the  expertise  of  the  interviewees  in  order 

to  explore  the  practicalities  of  the  proposed  theories. 
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Research  Objectives 


The  objective  of  each  phase  of  this  study  is  as  follows 

Phase  X.  To  examine  the  existence  of  the  reliability 
gap  in  tactical  airborne  radars  on  Air  Force  and  Navy 
aircraft. 

Phase  II.  To  identify  and  analyze  the  major 
contributors  to  the  reliability  gap.  Specifically,  to 
identify  the  moot  significant  contributors. 

Phase  III.  To  explore  ways  on  how  to  minimize  the 
reliability  gap.  Specifically,  to  identify  some  practical 
guidelines  on  reliability  growth. 


Research  Questions 

The  questions,  each  pertaining  to  its  respective 
objective  of  this  study,  are  as  follows: 

1.  Does  the  reliability  gap  exist  in  tactical  airborne 
radars? 

2.  What  are  the  major  contributors  to  the  reliability 
gap?  Specifically,  what  is  the  relative  importance 
of  the  major  contributors  and  what  are  the  most 
significant  contributors? 

3.  what  are  some  of  the  practical  guidelines  on  how  to 

minimize  the  reliability  gap?  Specifically,  what 
are  some  of  the  practical  prediction  techniques  and 
some  practical  guidelines,  i.e.,  the  growth  model, 
the  initial  reliability,  the  growth  rate,  and  the 
effectiveness  factor? 


Potential .Research  Benefits 

The  purpose  of  this  study  is  to  examine  and  analyze  the 
reliability  gap  between  the  field  and  factory  reliability. 
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In  addition,  this  thesis  identifies  some  practical 
guidelines  in  the  area  of  reliability  growth.  This  study 
offers  the  following  potential  benefits,  with  a  common  goal 
of  maximizing  operational  readiness  and  optimizing  life 
cycle  costs. 

1.  The  identification  of  the  most  significant 
contributors  to  the  reliability  gap  can  help 
decision  makers  to  better  allocate  their  resources. 

2.  The  comparative  (DoD  vs  Industry)  perspective  of  the 
most  significant  contributors  can  serve  as  a  guide 
to  better  understanding  the  reliability  gap. 

3.  This  research  suggests  some  possible  solutions  to 
the  most  significant  contributors  which  can  be 
implemented  in  the  future  to  minimize  the 
reliability  gap. 

4.  This  study  contains  an  extensive  literature  review 
on  the  issues  of  the  reliability  gap  and 
traditional  ways  of  reliability  assessment. 

5.  This  study  consolidates  some  of  the  best  currently 
available  thinking  on  the  major  contributors  to  the 
reliability  gap  and  some  of  the  practical 
guidelines  on  reliability  growth. 


Overview 

The  remaining  chapters  of  this  thesis  will  describe  in 
detail  how  the  objective  of  each  phase  of  this  study  will  be 
accomplished.  Chapter  II  presents  an  exhaustive  literature 
review  on  the  issues  of  reliability  as  they  pertain  to  this 
research,  while  Chapter  III  describes  the  data  requirement, 
data  collection,  and  the  data  analysis  of  this  research. 
Chapter  IV  presents  the  findings  and  Chapter  V  summarizes 
this  research  effort  and  makes  recommendations  for  future 
studies . 
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Research  of  this  study  begins  with  the  investigation  of 
the  reliability  gap  between  the  factory  (whether  specified, 
predicted,  or  demonstrated)  and  the  field  reliability.  This 
chapter  begins  with  the  definition  of  reliability,  followed 
by  the  development  of  reliability  engineering  and  the 
understanding  of  the  problems  of  unreliability.  Description 
of  the  reliability  gap  and  traditional  ways  of  assessing 
reliability  are  examined.  In  addition,  some  practical 
questions  and  answers  associated  with  the  research 
problem — the  reliability  gap— are  also  presented.  This 
chapter  concludes  with  a  summary  of  the  literature  review. 

Definition  of  Reliability 

In  order  to  have  a  clear  understanding  of  the  theme  of 
this  study — the  reliability  gap — it  is  first  very  important 
to  understand  the  meaning  of  the  term  called  "reliability." 

Reliability  is  defined  as: 

...the  probability  that  a  system  or  product  will 
perform  in  a  satisfactory  manner  for  a  given 
period  of  time  when  used  under  specified 
operating  conditions.  [8:14] 
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PROBABILITY,  the  first  element  in  the  reliability 
definition,  implies  that  any  attempt  to  quantify  reliability 
involves  the  use  of  statistical  methods.  Usually,  it  is 
quantified  in  terms  of  the  relative  frequency  of  occurrence 
of  an  event,  that  is,  the  ratio  of  the  number  of  times  that 
the  event  occurs  (successes)  to  the  total  number  of 
opportunities  for  occurrence  (trials)  of  the  event  (35:74), 
For  example,  the  probability  of  survival  of  an.  item  for  80 
hours  is  0.75  indicates  that  "we  can  expect  the  item  will 
function  properly  for  at  least  80  hours,  75  times  out  of  100 
trials"  (8:14).  It  is  also  important  to  realize  that  the 
observed  probabilistic  quantities  of  a  certain  parameter  are 
not  the  true  values  of  that:  parameter,  but  are  values 
distributed  around  some  expected  value  (33:111). 

SATISFACTORY  PERFORMANCE,  the  next  element  in  the  basic 

reliability  definition,  indicates  that: 

. . .  specific  criteria  must  be  established  which 
describe  what  is  considered  to  be  satisfactory 
system  operation.  A  combination  of  qualitative  and 
quantitative  factors  defining  the  functions  that 
the  system  or  product  is  to  accomplish,  usually 
presented  in  the  context  of  a  system  specification, 
are  required.  (8:15] 

TIME,  the  third  element  of  the  reliability  definition, 

is  recognized  as  one  of  the  most  important  elements  in  the 

basic  definition  of  reliability  since  it  represents: 

...a  measure  against  which  the  degree  of  system 
performance  can  be  related.  One  must  know  the 
’time'  parameter  in  order  to  assess  the  probability 
of  completing  a  mission  or  a  given  function  as 
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scheduled.  Of  particular  interest  is  being  able  to 
predict  the  probability  of  an  item  surviving 
(without  failure)  for  a  designated  period  of  time. 
Also,  reliability  is  frequently  defined  in  terms  of 
mean  time  between  failure  (MTBF) ,  mean  time  to 
failure  (MTTF)  or  mean  time  between  maintenance 
(MTBM) ;  thus,  the  aspect  of  time  is  critical  in 
reliability  measurement.  [8:15] 

Time  seems  to  be  a  relatively  simple  parameter;  some  of 

the  underlying  complexities  of  time  are  explained  by  Shelly 

and  Stovall  (62)  in  these  words: 

The  '  T'  part  of  MTBF  in  the  laboratory  might  be 
equipment  on  chamber-time,  official  test-time,  or 
various  combinations.  The  ' T'  in  the  field  MTBF 
might  be  flight  time,  equipment  operating  time, 
equipment  standby  plus  operating  time,  or  various 
combinations.  [62:322] 

SPECIFIED  OPERATING  CONDITIONS,  the  fourth  element  of 

the  basic  reliability  definition,  include: 

...environmental  factors  such  as  the  geographical 
location  where  the  system  is  expected  to  operate, 
the  operational  profile,  temperature  cycles, 
humidity,  vibration,  shock,  and  so  on.  Such 
factors  must  not  only  address  the  conditions  for 
the  period  when  the  system  or  product  is  operating, 
but  the  conditions  for  the  periods  when  the  system 
(or  a  portion  thereof)  is  in  a  storage  mode  or 
being  transported  from  one  location  to  the  next. 
Experience  has  indicated  that  the  transportation, 
handling,  and  storage  modes  are  sometimes  more 
critical  from  a  reliability  standpoint  than  the 
conditions  experienced  during  actual  system 
operational  use.  [8:15] 

In  determining  systems  or  products  reliability,  all  these 
elements  are  essential.  By  examining  the  basic  definition 
of  reliability,  it  can  be  realized  that  the  study  of 
reliability  is  rather  complex.  The  complexity  involves 
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the  uncertainties  associated  with  the  study  of  probability 
and  statistical  analysis. 


The  study  of  statistical  analysis  is  concerned  with 
the  "collection,  organization,  and  interpretation  of  data 
according  to  well-defined  procedures"  (35:3).  Well-defined 
in  the  sense  that  the  rules  of  organization  and 
interpretation  are  clearly  spelled  out  so  that,  given  the 
same  data,  any  researcher  applying  the  same  analytical 
techniques  will  get  the  same  conclusions  (35:10). 

However,  the  application  and  interpretation  of 
statistics  in  reliability  are  not  as  straight  forward  as 
public  opinion  polls  or  measurement  of  human  IQ  scores.  In 
these  applications: 

...most  interest  is  centered  around  the  behaviour 
of  the  larger  part  of  the  population  or  sample, 
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In  reliability  we  are  concerned  with  the  behaviour 
of  unlikely  combination  of  load  and  strength, 
variability  is  often  hard  to  quantify  and  data  are 
expensive,  [58:4] 

Reliability  analysis  may  contain  many  sources  of 
uncertainty  and  variability  which  may  be  more  difficult  to 
analyze  than  other  disciplines  of  statistical  analysis. 
However,  O'Connor  (58)  emphasized  how  people  can  make  a 
difference  in  reliability  analysis: 


...the  reliability  engineer  or  manager  is  not,  like 
an  insurance  actuary,  a  powerless  observer  of  his 
statistics.  Statistical  derivations  of  reliability 
are  not  a  guarantee  of  results,  and  these  results 
CAN  be  significantly  affected  by  actions  taken  by 
quality  and  reliability  engineers  and  managers. 
[58:5] 
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Reliability  is  defined  in  terms  of  four  essential 
elements:  Probability,  satisfactory  performance, 

time,  and  specified  operating  conditions.  Knowing  the 
essential  elements  of  reliability  is  important  in 
understanding  the  research  problem— —the  reliability  gap. 

The  study  of  reliability  has  been  in  existence  for  quite 
a  number  of  years.  In  fact,  the  study  of  reliability 
analysis  has  its  roots  back  in  the  1940s,  during  WWII 
(37:5) . 

Development  of  Reliability  Eng  in  goring 

Reliability  analysis  is  "an  indirect  outgrowth  of  the 
problems  with  electronic  systems  designed  during  the  early 
1940s  for  use  in  the  war  effort"  (37:5).  Over  the  years, 
there  seemed  to  be  a  direct  relationship  between  the 
complexity  of  military  electronic  systems  and  the  problems 
that  were  generated.  As  we  transitioned  into  the  age  of 
solid-state  electronics  technology,  the  problems  were 
compounded  oy  new  designs,  new  manufacturing  processes,  and 
perplexing  logistics  support  issues.  All  these  problems 
translated  into  reduced  availability  and  increased  ownership 
costs  (58:9) . 

In  1952,  the  Department  of  Defense  (DoD)  established  the 
Advisory  Group  on  Reliability  of  Electronic  Equipment 
(AGREE)  to  investigate  the  reliability  problem  (37:5).  In 
1955,  the  AGREE  established  a  program  consisting  of  nine 
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tasks:  numerical  reliability  requirements,  tests,  design 
procedures,  components,  procurement,  packaging  and 
transportation,  storage,  operation,  and  maintenance.  The 
task  groups,  comprised  of  personnel  from  both  the  military 
and  industry’,  ’’were  asked  to  submit  their  findings  in  the 
form  of  a  report  after  they  had  considered  all  aspects  of 
their  assigned  tasks"  (1).  In  the  area  of  testing,  the  task 
group  generated  a  report  which  concluded  that  integral 
activities  are  needed  in  the  development  cycle  in  order  to 
break  the  spiral  of  increasing  development  and  ownership 
costs.  The  report  emphasized  the  need  to  test  new  equipment, 
for  several  thousand  hours  in  high-stress,  cyclical 
environments  in  order  to  discover  design  problems  early  so 
they  can  be  corrected  before  production.  The  report  also 
recommended  that  formal  demonstrations  of  reliability,  using 
statistical  methods,  be  instituted  as  an  acceptance  condition 
of  equipment  by  the  procuring  agency.  The  DoD  accepted  and 
reissued  the  AGREE  report  on  testing  as  US  Military  Standard 
(MIL-STD)  781,  Reliability  Qualification  and  Production 
Approval  Tests.  The  reliability  engineering  effort 
developed  quickly.  The  AGREE  and  reliability  program 
concepts  were  soon  adopted  by  the  National  Aeronautics  and 
Space  Administration  (NASA)  and  many  major  companies 
(58:10) . 


In  1965,  the  DoD  issued  MIL-STD-785,  Reliability 
Programs  for  Systems  and  Equipment.  This  document  mandated 
the  integration  of  reliability  engineering  activities  with 
the  traditional  engineering  design,  development,  and 
production  activities.  It  was  realized  that  an  integrated 
approach  was: 

...the  only  way  to  ensure  that  potential 
reliability  problems  would  be  detected  and 
eliminated  at  the  earliest,  and  therefore  the 
cheapest,  stage  in  the  development  cycle.  [58:10] 

Numerous  studies  were  conducted  to  show  the  cost  benefit  of 

higher  reliability  as  a  result  of  early  development  effort, 

demonstration  of  specified  levels  of  reliability  in 

accordance  with  MIL-STD-781,  and  production  testing.  The 

concept  of  life  cycle  cost  was  introduced  as  part  of  the 

cost  benefit  analysis  (58:10). 

As  the  struggle  for  more  reliable  systems  or  products 
continues,  there  is  a  need  to  better  understand  reliability. 
The  next  section  will  address  the  reasons  for  unreliability 
and  the  problems  associated  with  it. 


As  far  back  as  1955,  in  the  early  years  of  reliability 
engineering,  RCA  conducted  a  study  and  presented  the 
findings  in  the  Proceedings  of  the  Conference  on  "The 
Reliability  of  Military  Electronic  Equipment,"  August  1956. 
The  summary  of  the  study  contained  10  major  conclusions 
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about  reliability  and  Appendix  A  contains  7  of  the  10 
conclusions.  What  is  so  amazing  is  that  most  of  the 
conclusions  still  apply  today,  33  years  later!  Especially 
interesting  is  the  conclusion  that  "RELIABILITY  TAKES 
TIME.... time  is  the  mortar  which  binds  this  whole  structure 
of  reliability  together...."  (Appendix  A). 

Like  vintage  wine,  reliability  takes  time.  The  problems 
we  have  had  with  unreliability  over  the  years,  is  it 
possible  that  we  have  been  "drinking  the  wine  before  its 


time?" 

There  are  many  reasons  for  unreliability.  Lloyd  and 

Lipow  (14)  stated  that  the  root  cause  of  the  problems  is 

"due  to  the  dynamic  complexity  of  system  development 

concurrent  with  a  background  of  urgency  and  budget 

restrictions"  (41:3).  The  dynamic  complexity  to  which  they 

refer  is  the  emerging  technology  in  an  ever-changing 

environment.  They  noted  that: 

...devices  and  systems  are  not  perfect;  they  do 
not  operate  in  the  same  manner  in  all 
circumstances.  Our  total  knowledge  may  be 
insufficient  about  any  item  so  that  when  it  is 
placed  into  an  environment,  about  which  we  also 
have  insuf f iciency  of  information,  failure  occurs. 
{41:4] 

However,  they  mentioned  that  if  sufficient  time  is 
given,  we  will  acquire  the  knowledge  to  change  the  situation 
through  the  process  of  evolution.  They  expressed  the 
problem  in  these  words: 
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Were  we  able  to  let  the  evolution  take  place  at  a 
natural  pace,  our  reliability  problem  might  be 
relatively  minor.  Unfortunately,  there  is  an 
urgency  which  prevents  us  from  giving  sufficient 
time  to  all  of  the  many  considerations.  The 
evolutionary  process  conflicts  with  the 
'revolutionary1  atmosphere.  Before  we  have  time  to 
experience,  synthesize,  and  apply  our  knowledge  we 
are  developing  another  system  or  device.  [41:4] 

To  help  understand  the  problem  of  unreliability, 

Stovall  (65)  made  the  following  conclusions: 

- The  consumer  desires  a  higher  reliability  than: 

(a)  he  is  willing  to  pay  for,  or 

(b)  he  knows  how  to  obtain  from  the  producer. 

- The  manufacturer  is  frequently  unwilling  to 

spend  the  money  necessary  to  achieve  high 
reliability  because  of  zero  profit  margin 
brought  about  by  the  low  quote  necessary  to  get 
the  business. 

— -Cost/schedule  is  a  trade-off  against  the 

achievement  of  high  reliability;  reliability 
is  usually  compromised  with  both  consumer  and 
producer  being  parties  to  the  decision  to 
establish  cost/schedule  as  the  number  one 
priority. 

- More  often  than  not,  the  supplier  does  not  know 

how  to  achieve  higher  reliability  than  he 
achieved  in  the  past,  and  he  does  not  have  the 
reliability  management  skill  to  elevate  his 
capability. 
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levels:  prime  contractor,  supplier,  and  sub¬ 
tler  supplier.  Although  the  same  may  be  said 
for  the  design,  it  is  easier  to  design  high 
reliability  than  it  is  to  produce  it. 

* — Fcr  high  piece-part  count  equipment,  subsystems 
ara  bo  complex  with  so  many  interfaces  with 
other  subsystems  that  there  is  an  inherent  low 
reliability. 


-—Frequently,  the  operational  facility  does  not 
.*ave  the  necessary  maintenance/repair 
capability  for  complex  systems  to  sustain 
original  hardware  reliability. 

- Reliability,  as  delivered  from  the  producer  is 

frequently  directly  related  to  the  future 
business  potential.  Today's  action  is  strongly 
influenced  by  the  potential  for  tomorrow's 
sale.  [65:594] 

In  the  early  1980s,  Air  Force  Logistics  Command  (AFLC) 
did  a  study  on  the  cost  of  unreliability.  The  study  found 
that: 

...parts  failures  accounted  for  75%  of  support 
equipment  costs  in  aircraft  procurement  accounts 
and  at  least  20%  of  the  Air  Force  budget.  The 
study  also  showed  that  the  impact  of  improving 
reliability  was  significant.  In  fact,  for  a 
composite  of  fighter  aircraft,  doubling  the  mean 
time  between  failure  (MTBF)  would  reduce  the  spares 
requirement  by  some  80%.  [32:5] 

Reliability  is  not  a  simple  concept.  It  is  defined  in 
termr  of  four  essential  elements:  probability,  satisfactory 
performance,  tune,  and  specified  operating  conditions. 
Reliability  anc lysis/engineering  is  not  a  new  science 
(though  some  may  still  consider  it  an  art) ;  it  has  been  in 
existence  since  the  1940s.  Over  the  years,  there  have  been 
numerous  studies  on  the  puzzling  concepts  of  reliability/ 
unreliability.  With  the  stage  set,  this  study  will  now 
explore  the  research  problem — the  reliability  gap. 


The  Reliability  Gap 

The  inability  to  relate  the  factory  (whether  specified, 
predicted,  or  demonstrated)  and  field  reliability  has  proven 
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to  be  costly.  It  has  been  shown  that  the  "inaccurate 
predictions  of  reliability  characteristics  may  result  in 
non-optimum  allocation  of  program  resources,  and,  in  turn, 
low  level  of  operational  readiness"  (40:245) .  Before 
examining  the  composition  of  the  reliability  gap,  it  is 
necessary  to  define  the  reliability  indices  used  to  measure 
reliability.  In  his  study,  Kern  (Hughes  Aircraft)  described 
the  terms  as  follows: 


The  required  MTBF  is  established  on  considerations 
of  mission  requirements,  cost,  previous  experience, 
etc.  The  predicted  MTBF  is  an  analytical 
assessment  of  the  inherent  reliability  based  on 
equipment  design  characteristics,  while 
demonstrated  MTBF  assess  inherent  reliability  on 
the  basis  of  the  failure/ operating  time  experience 
of  a  specific  sample  of  equipment  under  controlled 
laboratory  test  conditions.  The  field  MTBF,  on  the 
other  hand,  is  an  assessment  of  the  achieved 
operational  reliability  of  the  equipment  in  actual 
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Kern  addressed  that  in  theory,  since  all  four  MTBFs 


measure  the  same  parameter,  they  should  be  alike.  But  in 


practice: 


...  it  has  been  reported  that  the  ratio  of  an 
equipment's  demonstrated  MTBF  to  its  field  MTBF  can 
range  from  7:1  to  20:1.  Even  greater  disparities 
are  ouoted  when  comparing  predicted  MTBFs  to  field 
MTBFs.  [38:231] 

With  four  types  of  reliability  indices,  there  are  six 
possible  combinations  of  reliability  gap:  Predicted  vs 
Specified,  Predicted  vs  Demonstrated,  Predicted  vs  Field, 
Specified  vs  Demonstrated,  Specified  vs  Field,  and 
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Demonstrated  vs  Field.  Figure  2  presents  a  graphic 
description  of  the  reliability  indices  for  two  typical 
systems . 


PREDICTED 

PREDICTED 

DEMONSTRATED 

SPECIFIED 

(LABORATORY) 

(REQUIRED) 

SPECIFIED 

DEMONSTRATED 

(REQUIRED) 

(LABORATORY) 

FIELD 

FIELD 

SYSTEM  A 


SYSTEM  B 


Figure  2,  The  Reliability  Indices 
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The  following  paragraphs  will  address  the  reliability 
gap  of  each  combination.  In  some  cases,  the  contributing 
factors  of  the  reliability  gap  for  the  particular 
combination  may  be  described  under  the  same  paragraph. 


Predicted  vs  Specified  vs  Demonstrated  vs  Field  MTBF . 
Since  some  of  the  factors  are  similar,  this  paragraph  will 
also  describe  the  factors  under  the  combinations  of 
Predicted  vs  Field  and  Specified  vs  Demonstrated.  Kern 
examined  16  different  pieces  of  avionics  equipment  from  10 
different  IJSAF  aircraft  and  attributed  the  differences  to 
"definitional  factors,  operational  factors,  or  environmental 
factor."  After  removal  of  the  definitional  factors,  he 
concluded  that: 

...about  half  of  the  remaining  differences  are 
accounted  for  by  operational  factors  and  the 
remainders  are  due  to  the  combined  influence  of 
environmental  and  other  factors.  [38:231] 

Kern  described  the  "definitional  factors"  as: 

...inherent  to  the  differences  in  the  failure 
criteria  and  time  base  used  by  the  two 
communities:  the  AFLC  which  collects  and 

analyzes  ulic  data  j  anu  the  eiiy ifieeirliig  Community 

(AFSC  and  Industry)  which  establishes  requirements, 
performs  predictions,  and  conducts  reliability 
demonstration  tests.  The  definitional  differences 
are  composed  of  two  parts,  one  related  to  the  time 
base  used  for  MTBF  assessment,  the  other  to  the 
failure  criteria  used  for  assessment  purposes. 
[38:241] 
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Kern  cited  that  the  primary  "operational  factors"  are: 

...those  related  to  maintenance  and  handling  and 
equipment  use.  The  study  shows  that  non¬ 
operating  failures  make  a  significant 
contribution  to  the  assessed  field  MTBF  for 
avionics  equipment.  It  is  estimated  that  between 
20%  and  60%  of  the  failures  recorded  during 
operational  deployment  of  avionics  equipment  are 
actually  non-operating  failures.  This  suggests 
the  need  to  establish  several  separate  measures  of 
MTBF,  each  directed  at  a  specific  objective.  One 
would  be  used  to  determine  the  inherent  reliability 
(based  on  operating  hours) ,  one  to  determine  field 
operational  reliability  (based  on  flight  hours) , 
and  another  to  determine  logistic  support 
reliability  (based  on  calendar  time,  i.e.,  months 
or  years).  [38:241] 

Under  "environmental  factors",  Kern  observed  that: 

. . .when  the  field  MTBF  data  are  separated  by 
aircraft  type  (fighters  vs.  bombers),  the 
differences  in  assessed  field  operational  MTBFs 
suggest  that  it  may  not  be  valid  to  use  a  single 
environmental  factor  for  aircraft  without  regard  to 
the  type  of  aircraft.  Possibly  the  currently 
published  environmental  factors  given  in  MIL  HDRK- 
217B  should  be  adjusted  by  an  appropriate  modifier 
to  reflect  differences  in  aircraft  type.  [38:241] 


Predicted  vs  Specified  MTBF.  Muglia,  et  al.,  (56) 
studied  the  reliability  gap  by  comparing  MTBF  predictiow 
techniques  (including  those  in  MIL-HDBK-217  A/B)  and  based 
the  analysis  on  13  years  of  actual  data  analysis  from 
Hoffman  Electronics  Corporation's  (HEC)  reliability  experience 
of  Tactical  Air  Navigation  (TACAN)  systems.  The  study 
concluded  that  the: 

...estimated  system  Mean-Time-Between-Failure 
(MTBF)  can  vary  from  0.16  to  6.5  times  that 
specified.  The  method  for  estimating  MTBF  is 
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responsible  for  this  variance-  Customer 
specifications,  the  engineer's  experience,  and 
management  desires  often  dictate  the  MTBF 
prediction  techniques  to  be  used.  [56:510] 

Muglia's  study  emphasized  further  that: 

System  prediction  techniques  become  a  relatively 
ineffective  guide  to  product  reliability 
measurement  if  management  does  not  support  the 
reliability  engineering  concepts.  More 
specifically,  management  must  allow  time  and  funds 
for  testing,  repair,  and  design  correction  where 
necessary  to  eliminate  thermal  and  electrical 
design  deficiencies,  manufacturing  deficiencies, 
and  testing  and  handling  deficiencies.  [56:513] 


Predicted  vs  Demonstrated  MTBF.  When  Lynch  and  Phaller 
(43)  examined  the  disparity  between  predicted  vs  test 
(demonstrated)  MTBF,  they  studied  an  Electronic 
Countermeasures  (ECM)  system  consisting  of  four  major  units 
containing  5,527  electronic  parts  for  seven  years.  Based  on 
their  study,  Lynch  and  Phaller  concluded  that  certain 
assumptions  inherent  in  the  MIL-HDBK-217  prediction  models 
were  largely  responsible  for  the  difference.  Specifically, 
the  assumptions  made  at  the  part  level  have  a  significant 
impact  on  the  reliability  prediction  and  they  found  that  the 
largest  areas  of  disparity  between  predicted  MTBF  and  test- 
observed  MTBF  were  the  assumptions  made  for: 

(1)  the  quality  of  design  and/or  constructions 
techniques  used  in  initially  designing  am’ 
laying  out  the  parts,  and 

(2)  the  adherence  to  the  established  and  specified 
production  process  control  procedures  in 
producing  the  parts.  In  the  models  presented 
in  MIL-HDBK-217,  it  is  assumed  that  certain 
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standards  are  followed  in  these  areas  based  on 
the  requirements  specified.  The  adherence  to 
these  requirements  manifest  themselves 
primarily  in  the  MIL-HDBK-217  component 
quality  factor  which  is  subject  to  wide 
variation  (120  to  1  for  hybrids) .  There  are 
also  equivalent  assumptions  regarding  system 
design  and  quality  practices  which  can  affect 
this  disparity....  [43:121] 


Demonstrated  vs  Field  MTBF.  In  1974,  the  "Joint 
Logistics  Command  (JLC)  sponsored  a  Reliability  Test 
Committee  as  part  of  its  Electronics  Systems  Reliability 
Workshop"  (2:30).  The  Committee  was  comprised  of 
experienced  reliability  personnel  from  both  the  Department 
of  Defense  (DoD)  &  Industry.  Their  task  was  to  better 
understand  the  underlying  causes  of  the  discrepancies 
between  laboratory  and  field  reliability  in  electronic 
systems.  The  Committee  classified  their  findings  into  four 

gnncra  1  CSftSCjOrisS  I 

- DIFFERENCES  IN  FAILURE  DEFINITIONS.  In 

laboratory  testing,  the  basic  reliability 
measure  is  derived  from  relevant  failures,  those 
that  cause  loss  of  function  in  the  equipment 
under  test,  in  operational  use,  the  basic 
measure  is  maintenance  action,  which  may  not  be 
related  to  the  loss  of  function  in  the  equipment 
being  investigated,  but  is  induced  by  an 
external  source?  thus,  there  may  be  more 
maintenance  actions  than  relevant  failures . 

- POOR  DEFINITION  OR  SIMULATION  OF  THE  OPERATIONAL 

USE  ENVIRONMENT.  Present  laboratory  testing  is 
oriented  heavily  toward  temperature 
environments,  with  some  consideration  to 
vibration  and  voltage  cycling.  In  actuality, 
humidity  is  important  to  seaborne  systems, 
random  vibration  to  jet  aircraft,  and  shock  to 
army  mobile  units,  yet  these  environments  are 
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not  specified  for  laboratory  reliability 
testing.  Unless  these  factors  have  been 
carefully  considered  in  design,  the  testing  is 
inadequate. 


■MISINTERPRETATION  OF  TEST  PLAN  RESULTS.  Many  of 
the  present  test  plans  in  the  existing  standards 
are  designed  to  efficiently  demonstrate  a 
minimum  acceptable  MTBF  at  high  confidence 
levels.  However,  the  confidence  with  which  the 
specified  MTBF  is  demonstrated  is  a  variable 
dependent  on  the  test  length  and,  therefore,  is 
a  cost-of-testing  consideration.  Unless  the 
user  understands  the  difference  between  these 
two  parameters,  unforeseen  results  could  occur. 


- INADEQUATE  FAULT  ISOLATION  TECHNIQUES.  Existing 

military  electronic  systems  have  grown  very 
complex.  But  the  gains  in  technology  have  been 
devoted  to  increased  performance,  with  inadequate 
emphasis  being  placed  on  design  for  ease  of 
maintenance.  When  failures  later  occur  in 
operational  use,  fault  isolation  becomes 
extremely  difficult.  Consequently,  the 
maintenance  personnel  under  pressure  of 
operational  readiness  are  forced  to  resort  to 
cannibalization  and  'shotgun'  techniques.  The 
penalty  is  maintenance- induced  faults  and  large 
percentages  of  'no  defect  found'  in  equipment 
returned  to  depots.  However,  the  added  cost  for 


this  fault  isolation  capability  in  the  design  is 
a  deterrent  in  the  acquisition  phase  of  a 
competitive  environment .  [2:31] 


Shelley  and  Stovall  also  studied  the  relationship 
between  the  field  and  laboratory  MTBFs.  By  using  various 
adjustment  factors,  they  introduced  the  translational  model 


to  establish  a  mathematical  relationship  between  field  and 


demonstrated  reliability.  They  concluded  that: 

(1)  It  is  not  possible  to  consistently  predict  the 
field  MTBF  from  laboratory  data,  and  vice 
versa.  The  individual  predictions  will 
invariably  be  high  or  lew.  However,  if  one 
counts  all  laboratory  discrepancies  (except 
those  due  to  test  equipment) ,  there  is  an  even 
chance  that  the  field  MTBF  will  be  greater 
than  the  laboratory  MTBF. 
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(2)  Prediction  accuracy  and  correlation  are  greatly 
improved  when  the  laboratory/ field  MTBF 
numbers  are  based  upon  several  failures,  i.e., 
statistical  fluctuation  is  a  real  hazard.  One 
can  seldom  be  accurate  by  attempting  to 
project  field  performance  from  lab  performance 
when  the  laboratory  MTBF  is  based  on  a  few 
(say  2  or  less)  failures,  and  conversely. 

(3)  A  major  reason  for  differences  between 
laboratory  and  field  reliability  is  failure 
definition  combined  with  the  human  element. 

In  the  field,  there  is  a  strong  inclination  to 
count  discrepancies  as  failures  unless  if  is 
very  obvious  that  there  was  no  failure.  In 
the  laboratory,  there  is  a  strong  inclination 
to  exclude  a  discrepancy  unless  it  is  very 
obvious  that  there  is  a  failure  as 
contractually  defined. 

(4)  The  relatively  good  correlation  of  reject  data 
and  the  very  poor  correlation  of  accept  data 
provides  strong  evidence  that  the  problem  of 
poor  correlation  is  more  people-related  than 
test  requirement-related. 

(5)  It  was  not  conclusively  shown  that  tne 
translation  between  laboratory  and  field 
MTBF ' s  can  be  improved  by  the  use  of 
adjustment  factors,  as  determined  in  this 
study.  The  translation  model  was  not 
disproved,  but  it  is  apparent  that  betcer 
information  is  needed  in  order  to  confidently 
utilize  the  model.  [62:330] 

In  addressing  the  difference  between  field  reliability 

and  factory  reliability,  whether  specified,  predicted,  or 

demonstrated  values  (44:177),  MacDia.*mid  (Rome  Air 

Development  Center)  mentioned  several  studies.  These 

studies  tried  to  clarify  the  issue  by  introducing: 

. . .the  notion  that  there  is  a  set  of  factory 
reliabiity  and  maintainability  terms  called 
•contract'  terms  and  a  set  of  field  R&M  terms 
called  'operational*  terms  that  are  defined 
differently  and  serve  different  purposes.  [44:177] 
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In  addition,  KacPiarmid  summarized  the  results  of  the 

Boeing  study  on  translating  operational  R&M  parameters  to 

contractual  ones,  and  vice  versa: 

The  Boeing  results  do  not  attempt  to  find  a  simple 
degradation  factor  relationship  between  a  field 
MTBF  and  a  contract  MTBF  as  others  have  often  done, 
but  instead  they  recognize  that  the  difference  in 
contractual  and  operational  parameters  are  in 
concept,  measurement  method,  and  usage  and  provide 
means  to  relate  them.  [44:182] 

In  trying  to  understand  tb.a  disparity  between  field  and 
factory  reliability,  Lynch  and  O' Berry  (42)  collected  over 
two  years  of  data,  observed  over  500  syscems  deployed  at  10 
operational  sites.  They  conducted  that  the  most  significant 
factor  affecting  field  reliability  is  "composed  of  much  more 
than  the  elements  of  temperature,  vibration,  altitude,  etc., 
that  are  simulated  in  a  typical  MIT -STD-781  test"  (42:242). 
They  found  that  the  environment  beyond  those  identified  in 
MTL-STD— 7 8 1  to  be  maintenance-  and  logistics  support-related 
and  it  includes : 

. . .maintenance  personnel,  shop  management,  program 
management,  support  equipment,  prime  system  and 
support  equipment  spares,  planning  and  operations, 
and  the  complex  interaction  among  them.... 

[43:242] 

From  their  field  experience,  they  also  found  that 
"...only  20%  of  the  field  problems  encountered  were  hardware 
reliability  problems"  (42:242). 


Balaban  (4)  identified  the  limitations  associated  with 
field  data  collection  systems  and  contributed  the  difference 
between  field  and  test  (demonstrated)  reliability  to  three 
classes; 


- Analysis  and  Test  Weaknesses.  During  the 

development  and  production  phases,  reliability 
engineers  must  deal  with  many  ambiguities  and 
make  many  assumptions  that  will  subsequently 
represent  weaknesses  in  their  efforts.  For 
analysis  effort,  these  include  modeling  an 
evolving  design  or  a  set  of  design  options; 
developing  assumptions  related  to  equipment 
interfaces,  environments,  built-in  test  (BIT) 
capabilities;  using  historical  data  derived  from 
incompatible  hardware,  software,  support,  or 
environmental  elements;  and  estimating  the 
nature  and  capability  of  the  equipment's  support 
environment.  Even  though  system  predictions  are 
based  on  the  best  data  available,  the  analyst 
works  under  numerous  limitations  that  affect  the 
quality  of  the  resulting  estimate.  For  test 
efforts,  the  results  must  be  tempered  by  the 
effects  of  in-process  and  subsequent  design 
changes;  the  use  of  nonrepresentative  hardware; 
and  test  environments  that  are  nonrepresentative 
or  lack  BIT  or  interface  software,  test 
equipment,  manuals,  or  appropriately  qualified 
operators  or  support  personnel.  There  is  always 
a  trade-off  between  test  length,  test  timing, 
and  test  realism.  These  compromises  limit  the 
quality  of  the  resulting  estimate. 

- Improper  Assumptions.  Independent  of  the 

problems  that  beset  the  analysis  and  test 
efforts....  This  can  include  changes  in  the 
physical  operating  environment  (e.g.,  the 
avionics  bay  is  or  is  not  vibration-free) ,  in 
usage  assumptions  (e.g.,  the  mission  length  is 
changed),  and  in  support  concepts  (e.g.,  the 
type  or  amount  of  test  equipment  to  be  used) . 

- Variability  of  Results.  Similar  hardware, 

operated  under  supposed  similar  conditions,  can 
exhibit  widely  varying  reliability 
characteristics.  This  variability  underscores 
the  difficulty  of  developing  point  estimates  of 
reliability  characteristics  during  development 
and  test  activities.  [4; 123] 
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The  existence  of 


the  reliability  gap  can  be  attributed  to  a  host  of  factors. 
A  summary  of  the  major  contributing  factors,  along  with 
their  references,  is  presented  in  Table  2. 


Table  2.  The  Reliability  Gap:  Literature  Review  summary 
Major  Contributing  Factors  References 


Definitional 

Operational ,  Usage 

Environmental 

Prediction  Techniques 
- — including  those  in  217 

Prediction  Assumptions 
- specifically  those  in  217 

Misinterpretation  of  Test  Plan 
Results 

Inadequate  Fault  Isolation 
Techniques 

Analysis  &  Test  Weaknesses 
Improper  Assumptions 
Variability  of  Results 
Reliability  Measurement  Methods 
Management  Support 
Statistical  Variability 
Human  Performance 


2,  29,  38,  44,  62 
2,  4,  38,  44 
2,  29,  38,  42 
56,  62 

4,  43,  62 

2 

2 

4 

4 

4 

44 

56 

62 

62 
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Traditional  Wavs  of  Assessing  Reliability 


The  development  of  highly  complex  weapon  systems 
involves  considerable  amount  of  risks.  The  ultimate  risk  is 
in  the  development  of  highly  complex,  state-of-the-art 
weapon  systems  that  do  not  meet  the  operational 
requirements.  In  order  to  minimize  the  risks,  various 
techniques  were  used  to  track  the  reliability  value 
throughout  the  development  process.  These  techniques  also 
help  decision  makers  to  pinpoint  specific  problem  areas 
early  in  the  development  phase,  make  better  decisions  in 
terms  of  trade-offs,  and  consequently  make  the  overall 
development  process  more  cost  effective. 

This  section  of  the  literature  review  is  divided  into 
three  portions:  Reliability  Prediction,  Reliability 
Testing,  and  Reliability  Growth.  The  basic  concepts  of 
each,  along  with  its  objectives,  are  presented  in  the  following 
paragraphs . 

Reliability  Prediction.  This  portion  addresses  the 
underlying  objective  of  reliability  prediction,  reliability 
models  (definition,  limitations,  and  applications) ,  and  the 
standard  references  for  reliability  prediction  and  the 
limitations  of  MIL-HDBK-217. 

Reliability  Prediction  Objectives.  The  underlying 
objectives  of  reliability  prediction  are  to  maximize 
operational  readiness  and  optimize  life  cycle  costs  (40:245). 
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In  addition,  making  a  reliability  prediction  is  also  used  to 
gain  advance  knowledge  of  the  reliability  of  a  new  system 
or  product.  The  advance  knowledge  of  reliability  "could 
allow  accurate  forecasts  to  be  made  of  support  costs,  spares 
requirements,  warranty  costs,  marketability,  etc."  (58:122). 
From  an  engineering  point  of  view,  reliability  prediction  is 
invaluable  as  part  of  the  design  processes  for  comparing 
options  and  to  identify  critical  reliability  design 
features  (58:122). 

Reliability  Models.  Various  reliability  models 
have  been  used  in  an  attempt  to  predict  the  reliability  of 
systems  and  subsystems  with  more  accuracy. 

Definition.  In  order  for  reliability  to  be 
accepted  as  a  science,  it  has  to  satisfy  two  conditions: 

1.  ...the  general  structure  of  its  formulae  must  be 
consistent  with  the  general  structure  of  the  other 
formulae  of  science  itself,  and 

2.  ...they  must  be  so  interpreted  to  yield  results  that 
can  be  confirmed  by  observations.  [33:110] 

Thus  mathematics,  with  its  formulae  serving  as  laws  or 
models,  has  been  used  as  the  language  of  reliability 
(33:110) . 

A  reliability  model  is: 

A  mathematical  relation,  that  on  the  basis  of 
deductive  or  inductive  reasoning  can  be  expected  to 
exist  between  some  measure  of  a  failure  process, 
and  one  (usually  time)  or  more  explanatory  factors. 
[33:111] 
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Limitations.  Since  reliability  is  defined  in 


terms  of  probabilities,  further  limitations  are  involved 

with  data  interpretation  and  sampling  variation.  As 

described  by  Harris  and  O'Connor: 

...in  dealing  with  probabilistic  quantities  we  no 
longer  consider  their  true  value  as  being  precisely 
predicted,  but  rather  as  being  distributed  around 
some  expected  value,  and  it  is  this  expected  value 
that  we  attempt  to  predict,  and  hopefully  to  gain 
some  insight  into  its  distribution...  [33:111] 

Another  limitation  is  the  assumption  of  the  use  of 
independent  and  identically  distributed  (IID)  random 
variables.  O'Connor  addressed  the  reasons  why  the 
assumption  of  IID  exponential  in  a  repairable  system  can  be 
very  misleading.  O'Connor  cited  a  total  of  14  reasons;  some 
of  the  reasons  are  as  follow  : 

1.  The  most  important  failure  modes  of  systems  are 
usually  caused  by  parts  which  have  failure 
probabilities  which  increase  with  time  (wearout 
failures) . 

2.  Failure  and  repair  of  one  part  may  cause  damage 
to  other  parts.  Therefore,  times  between 
successive  failures  ARE  NOT  NECESSARILY 
INDEPENDENT. 

3.  Repairs  often  do  not  'renew'  the  system. 

Repairs  are  often  imperfect  or  they  introduce 
other  defects  leading  to  failures  of  other 
parts . 

4.  Repair  personnel  learn  by  experience,  so 
diagnostic  ability  (i.e.,  the  probability  that 
the  repair  action  is  correct)  improves  with 
time.  Generally,  changes  of  personnel  can  lead 
to  reduced  diagnostic  ability  and  therefore 
more  reported  failures. 
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5.  Reported  failures  are  nearly  always  subject  to 
human  bias  and  emotion.  What  an  operator  or 
maintainer  will  tolerate  in  one  situation  might 
be  reported  as  a  failure  in  another,  and 
perception  of  failure  is  conditioned  by  past 
experience,  whether  repair  is  covered  by 
warranty ,  etc . 

6.  System  failures  might  be  caused  by  parts  which 
individually  operate  within  specification 
(i.e.,  do  not  fail)  but  whose  combined 
tolerances  cause  the  system  to  fail. 

7.  Many  reported  failures  are  not  caused  by  part 
failures  at  all,  but  by  events  such  as 
intermittent  connections,  improper  use, 
maintainers  using  opportunities  to  replace 
•suspect'  parts,  etc.  [58:89-91] 

An  overriding  criticism  about  the  concept  of  reliability 

modeling  is  the  one  involving  human  performance. 

Even  if  sufficient  failure  data  were  avi  liable  to 
overcome  the  deficiencies  of  statistical 
reliability  models  (an  impossible  situation  in 
the  practical  reliability  engineering  context) 
the  human  element  can  still  invalidate  the 
predictions  made.  [58:126] 

Dhillon  (24)  mentioned  that  H.  L.  Williams  was  the  first 


person  to  recognize  the  importance  of  human-element 
reliability  in  system-reliability  predictions  (1958). 

Dhillon  also  cited  the  study  of  A.  Shapero  and  pointed  out 
that  "human  error  is  the  cause  for  a  large  proportion  (i.e.. 


iron  v-u  au-ti 


)  of  all  equipment  failures"  (24:2). 


associated  with  reliability 
possible  and  useful  to  make 
systems  under  the  following 


Recognizing  all  the  limitations 
prediction  models,  it  is  still 
reliability  predictions  for 
circumstances : 
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1.  The  system  is  similar  to  systems  developed, 
built,  and  used  previously,  so  that  we  can  apply 
our  experience  of  what  happened  before. 

2.  The  new  system  does  not  involve  significant 
technological  risk  (this  follows  from  1) . 

3.  The  system  will  be  manufactured  in  large 
quantities,  or  is  very  complex  (i.e.,  contains 
many  parts) ,  or  will  be  used  over  a  long  time, 
or  a  combination  of  these  conditions  applies, 
i.e.,  there  is  an  asymptotic  property. 

4.  There  is  a  strong  commitment  to  the  achievement 
of  the  reliability  predicted,  as  an  overriding 
priority.  [58:127] 

For  example,  credible  reliability  predictions  can  be 
made  for  a  new  TV  receiver  or  cassette  player.  However,  for 
new,  highly  sophisticated,  state-of-the-art  equipment,  the 
reliability  predictions  are  not  as  credible  (58:127). 

Reliability  Predict^ on  Standards  References  and  the 
Limitations  of  MIL-HDBK-217.  Reliability  prediction  is  not 
a  simple  task  and  it  is  almost  an  impossible  task  for  new, 
highly  sophisticated,  state-of-the-art  equipment. 

MIL-STD-756,  Reliability  Prediction,  is  the  most  commonly 
used  standard  reference  for  reliability  prediction.  The 
prediction  plan  is  identified  as  part  of  the  reliability 
program  plan.  In  order  to  compare  options  and  validate  the 
concept,  MIL-STD-756  requires  that  an  initial  reliability 
prediction  be  performed  as  early  as  possible  as  part  of  the 
feasibility  study. 
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MIL-STD-217,  Reliability  Prediction  for  Electronic 
System,  is  the  standard  reference  for  electronic  equipment 
parts  count  and  stress  analysis  (58:150).  The  literature 
search  indicated  that  the  predictions  used  in  MIL-STD-217 
have  not  been  credible.  The  limitation  of  MIL-STD-217  is 
that  it  assumes  independent,  identically  exponentially 
distributed  times  to  failures  for  all  components.  However, 
O'Connor  measured  five  reasons  why  MIL-STD-217  is  acceptable 
in  most  work,  because: 

1.  A  constant  failure  rate  (CFR)  assumption  makes 
system  reliability  prediction  relatively  easy, 
since  an  additive  (parts  count)  method  can  be 
used. 

2.  For  most  maintained  equipment,  as  repairs  are 
carried  out  and  as  modules  and  components  are 
exchanged,  after  a  period  the  system  might 
consist  of  parts  with  different  ages.  Also, 
maintenance  induces  failures,  which  tend  to 
have  a  constant  rate  of  occurrence.  Therefore, 
there  might  be  an  overall  tendency  to  a  CFR. 

3 .  It  is  much  easier  to  calculate  an  assumed  CFR 
from  data  on  systems  than  to  derive  the 
parameters  of  a  two-parameter  (e.g.,  Weibull) 
distribution.  Also,  the  data  are  seldom 
numerous  enough  to  allow  derivation  with 
adequate  statistical  or  engineering  confidence. 

4.  For  logistics  planning  purposes,  e.g.,  avionics 
spares  planning  for  a  fleet  of  aircraft,  the 
CFR  model  is  adequate  both  for  prediction  and 
for  monitoring  of  performance.  MTBF  is  the 
function  usually  used  in  such  cases. 

5.  Predicted  reliability  is  subject  to  such  wide 
margins  of  error  that  the  IID/CFR  assumptions 
do  not  make  much  difference.  [58:189] 

Reliability  prediction  for  new,  highly  sophisticated, 
state-of-the-art  systems  must  be  based  on  the 
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"identification  of  objectives  and  assessment  of  risks,  in 
that  order"  (58:127).  Throughout  the  development  process, 
objectives  must  be  determined  to  see  whether  they  are 
realistic.  Appropriate  models  and  data  are  used  to  quantify 
the  objectives  and  assess  the  risks.  The  reduction  of  risks 
can  be  accomplished  through  analysis  and  tests  (58:128). 
Testing  is  crucial  to  the  development  of  complex  systems 
because  it  provides  decision  makers  with  the  feedback 
information  to  make  intelligent  trade-offs. 

Reliability  Testing.  This  portion  addresses  the 
objectives  of  reliability  testing.  Two  types  of  reliability 
testing,  reliability  demonstration  and  reliability 
development  testing,  will  be  discussed  in  detail. 

Reliability  Testing  Objectives.  The  objectives  of 
reliability  testing  is  to  reduce  the  risks  of  development  of 
highly  complex  systems  and  to  identify  the  reliability 
characteristics  as  early  as  possible  during  the  development 
program  because  "the  effect  of  failures  on  schedule  and  cost 
increases  progressively,  the  later  they  occur  in  the 
development  program"  (58:261).  Reliability  testing  is 
critical  in  the  development  of  complex,  highly  sophisticated 
systems  where  the  risks  are  high.  Testing  is  essential 
because  designs  are  seldom  perfect  and  "designers  cannot 
usually  be  aware  of,  or  be  able  to  analyze,  all  the  likely 
causes  of  failure  of  their  designs  in  service"  (58:260). 
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Demonstration  Testing.  It  is  also  Known  as 

Reliability  Qualification  Test  (RQT) ,  or  Design  Approval 

test  (55) .  It  is  defined  as: 

A  test  conducted  under  specified  conditions,  by, 
or  on  behalf  of,  the  government,  using  items 
representative  of  the  approved  production 
configuration,  to  determine  compliance  with 
specified  reliability  requirements  as  a  basis  for 
production  approval.  (Also  known  as  a 
'Reliability  Demonstration,'  or  'Design  Approval' 
test.)  [55:130] 

MIL-STD-781  is  the  standard  method  used  for  formal 
reliability  demonstration  testing.  MIL-STD-781  test  plans 
are  based  on  the  assumption  of  a  constant  failure  rate. 

The  practical  problems  associated  with  MIL-STD-781  testing 
are  as  follows: 

1.  RELIABILITY  IS  NOT  AN  INHERENT  PHYSICAL 
PROPERTY  OF  A  SYSTEM,  AS  IS  MASS  OR  ELECTRIC 
CURRENT.  The  mass  or  power  consumption  of  a 
system  is  measurable  (also  within  statistical 
bounds,  if  necessary) .  Anyone  could  repeat 
the  measurement  with  any  copy  of  the  system 
and  would  expect  to  measure  the  same  values. 
However,  if  we  measure  the  MTBF  of  a  system  in 
one  test,  IT  IS  UNLIKELY  that  another  test 
will  demonstrate  the  same  MTBF,  quite  apart 
from  considerations  of  purely  statistical 
variability.  In  fact,  there  is  no  logical  or 
physical  reason  to  expect  repeatability  of 
such  experiments....  Of  course,  if  a  large 
number. . .  were  tested  we  would  be  able  to 
extrapolate  the  results  with  rather  greater 
credibility  and  to  monitor  trends.... 

However,  MIL-STD-781  testing  cannot  be 
extended  to  such  large  quantities  because  of 
the  costs  involved. 

2.  MIL-STD-781  testing  is  often  criticized  on  the 
grounds  that  in-service  experience  of  MTBF  is 
very  different  to  the  demonstrated  figure.  In 
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addition,  in-service  conditions  are  almost 
always  very  different  of  the  environments  of 
MIL-5TD-781  testing,  despite  any  attempts  to 
simulate  realistic  conditions. 

3.  MIL-STD-781  testing  is  not  consistent  with  the 
reliability  test  philosophy...,  since  the 
objective  is  to  count  failures  and  to  hope 
that  few  occur.  An  effective  rellabiity  test 
programme  should  aim  at  generating  failures, 
SINCE  THEY  PROVIDE  INFORMATION  ON  HOW  TO 
IMPROVE  THE  PRODUCT.  Also,  a  reliability  test 
should  not  be  terminated  solely  because  more 
than  a  predetermined  number  of  failures  occur. 
MIL-STD-781  testing  is  very  expensive,  and  the 
benefit  to  the  product  in  terms  of  improved 
reliability  is  sometimes  questionable. 
[58:282-283] 


Development  Testing.  It  is  also  known  as 
Reliability  Development  Growth  Test  ( RDGT ) .  It  is  defined 
as : 


A  series  of  tests  conducted  to  disclose 
deficiencies  and  to  verify  that  corrective  actions 
will  prevent  recurrence  in  the  operational 
inventory.  (Also  known  as,  Test-Analyze-And-Fix, 

' TAAF '  testing).  [55:130] 

MIL-STD-781D  (53)  describes  how  development  reliability 
tests  should  be  managed  and  integrated  with  other 
development  tests. 

The  standard  was  developed  to  place  more  emphasis 
on  testing  to  detect  weaknesses  in  the  product, 
rather  than  on  formal  reliability  demonstration 
methods  in  which  the  test^pbjective  is  (from  the 
supplier’s  point  of  view)  to  have  as  few  failures 
as  possible.  [58:273-274] 

The  development  of  highly  complex,  state-of-the-art 
systems  involves  a  lot  of  uncertainties  and  risks.  In  order 
to  minimize  the  risks,  development  testing  programs  are 
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incorporated  to  ensure  that  system  reliability  and 
performance  requirements  are  met.  Over  the  years,  it  has 
been  recognized  that  "a  comprehensive  approach  to 
reliability  growth  management  throughout  the  development 
program"  (51:4),  is  necessary  to  minimize  the  high  risks 
associated  with  the  development  of  highly  complex  systems. 

Reliability  Growth  f Improvement' .  This  portion  of  the 
literature  review  addresses  the  basic  concepts  of 
reliability  growth,  reliability  growth  management 
(definition  and  the  managers'  role),  and  reliability  growth 
analysis  (purpose,  development,  selection  for  the  "optimum” 
growth  model,  and  application) . 

Basic  Concepts.  MIL-HDBK-189  defines  reliability 
growth  as  "the  positive  improvement  in  a  reliability 
parameter  over  a  period  of  time  due  to  changes  in  product 
design  or  the  manufacturing  process"  (51:3).  In  their 
article,  Morris  and  MacDiarmid  referenced  P.  H.  Mead's  study 
on  reliability  growth  of  electronic  equipment,  which  stated 
that  there  are  three  distinct  ways  in  which  reliability  can 
grow: 

Growth  Mode  1.  By  operating  each  equipment  (or 
portion  of  it)  to  expose  and  eliminate  rogue 
components  or  manufacturing  errors. 

Growth  Mode  2.  By  familiarization,  increased 
operator  skill  and  general  "settling  down"  in 
manufacturing,  use  and  servicing. 
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Growth  Mode  3 .  By  discovering  and  correcting 
errors  or  weaknesses  in  design,  manufacturing 
or  related  procedures.  [55:130] 

Morris  and  MacDiarmid  explained  further  that: 

Reliability  or  electronic  equipment  can  improve 
both  at  the  collective  and  individual  equipment 
level.  Burn-in  improves  the  reliability  of  the 
equipment  subjected  to  it  while  design  changes 
improves  (or  degrades)  the  reliability  of  all 
equipment  subject  to  the  changes.  Each  of  the 
three  growth  or  evolution  modes  can  be  made  more 
effective  bi  planned  activities.  [55:130] 

Reliability  growth  occurs  in  complex  systems;  however, 

the  misconceptions  concerning  reliability  growth  cannot  be 

overlooked.  Clarke  (14)  cited  cases  where  "reliability 

demonstration  test  data  have  been  improperly  used  to  portray 

reliability  growth"  (14:407).  According  to  Clarke: 

...to  effect  a  growth  in  inherent  reliability, 
one  or  more  of  the  basic  design  or  process 
(manufacturing  methods  or  quality)  parameters  must 
be  improved.  It  is  generally  recognized  that  a 
realistic  reliability  prediction,  based  upon  these 
parameters  is  a  good  approximation  of  the  inherent 
reliability  for  a  particular  design  and  the 
practical  upper  limit  for  reliability  growth. 
[14:407] 


Growth  Management.  KIL-HDBK-189  defines  it  as: 

The  systematic  planning  for  reliability  achievement 
as  a  function  of  time  and  other  resources,  and 
controlling  the  ongoing  rate  of  achievement  by 
reallocation  of  resources  based  on  comparisons 
between  planned  and  assessed  reliability  values. 
[51:3] 

It  is  emphasized  in  MIL-HDBK-189  that  the  various 
techniques  identified  in  reliability  growth  management  "do 
not,  in  themselves,  manage.  They  simply  make  reliability  a 
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To  ensure 


more  visible  and  manageable  characteristic." 
goals  are  achieved,  top  management  decisions  are  required 
to: 

- Revise  the  program 

— -Increase  testing 

- Fund  additional  development  effort 

- Add  or  reallocate  program  resources 

- Stop  the  program  until  interim  reliability  goals 

have  been  demonstrated  [51:5] 

Growth  Analysis.  Why  is  there  a  need  to  do 
reliability  growth  analysis?  Under  the  test-analyze-f ix 
process,  the  system  configuration  is  constantly  changing. 
Test  data  on  the  system  for  a  fixed  configuration  are 
limited.  Consequently: 

...direct  estimates  of  system  reliability  for  a 
fixed  configuration  would  generally  not  enjoy  a 
high  degree  of  confidence  and  may,  therefore,  have 
little  practical  value.  [18] 

Faced  with  the  difficulties  of  directly  estimating 
system  reliability,  reliability  growth  models  are  usually 
used.  Reliability  growth  models  are  defined  as  mathematical 
formulae,  usually  as  a  function  of  time,  used  to  represent 
the  system  reliability  during  the  development  phase.  The 
objectives  of  most  reliability  growth  models  are: 

(1)  Inference  on  the  present  system  reliability, 

(2)  Projection  on  the  system  reliability  at  some 
future  development  time.  [18] 

development  of  Growth  Model?.  During  the 
early  development  phases,  new  products  are  often  found  to  be 
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less  reliable  than  later  in  the  field,  when  failures  are 
discovered  and  improvements  have  been  made  and  incorporated 
as  a  result  of  failures  observed  and  corrected.  This 
phenomenon  of  displaying  reliability  improvement  (growth)  of 
products  in  service  was  first  analyzed  by  J.  T.  Duane  (25) . 
In  1962,  he  presented  a  report  on  the  empirical  relationship 
of  the  MTBF  improvement  he  observed  on  a  number  of  items 
used  on  aircraft.  Duane  observed  that  the  cumulative  MTBF 
(total  time  divided  by  total  failures)  plotted  against  total 
time  on  log-log  paper  gave  a  straight  line.  The  slope  of 
the  line  gives  an  indication  of  the  rate  of  MTBF  growth. 

The  steeper  the  slope,  the  faster  the  improvement,  and  the 
sooner  the  products  will  become  more  reliable.  Duane 
observed  that  the  typical  range  for  the  growth  rate  was 
between  0.2  and  0.4,  and  that  the  value  was  correlated  with 
the  intensity  of  the  reliability  improvement  effort 
(58:285-286) . 

In  addition  to  reliability  growth  models  for  hardware, 
some  researchers  have  also  explored  the  reliability  growth 
models  for  software.  In  his  study  on  reliability  growth 
models,  Balaban  mentioned  that  a  number  of  software 
reliability  models  have  been  developed  over  the  years. 

The  study  of  software  reliability  growth  models  is 
beyond  the  scope  of  this  research.  Unless  otherwise 
specified,  the  discussion  on  reliability  growth  model  in 
this  study  is  limited  to  hardware  only. 
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The  Optimum  Growth  Model.  Numerous  studies 
have  been  conducted  to  search  for  the  "ideal"  model.  Over 
the  years,  Rome  Air  Development  Center  has  conducted  and/or 
sponsored  several  studies  on  the  issue.  The  selection  of 
the  optimum  reliability  growth  model  is  beyond  the  scope  of 
this  research.  This  study  will,  however,  include  the 
findings  of  three  of  the  studies.  These  three  studies  were 
chosen  because  of  the  extensive  amount  of  research  effort 
that  was  involved. 

In  1975,  Schafer  (Hughes  Aircraft  Company)  examined  six 
models  and  fitted  270  data  sets  (186  ground  equipment  and  84 
airborne  equipment)  to  those  models.  The  findings  were: 

1.  The  results  indicate  that  although  the  Duane 
model  seldom  was  the  best  fitting  model  it 
almost  always  fit  the  data. 

2.  The  IBM  model  fit  airborne  data  the  best.... 

3 .  Each  of  the  remaining  models  was  found  to  be 
the  best  fit  to  the  data  for  specific 
combinations  of  environment,  equipment  type, 
and  aggressiveness  of  reliability  program. 

[61] 

In  1983,  McGlone  (Pratt  &  Whitney)  conducted  a  22-month 
program  to  study  and  analyze  the  reliability-growth 
phenomena  obtained  of  electronic  and  hydromechanical  gas 
turbine  control  equipment"  (49:53).  Five  mathematical 
models  were  evaluated  and  the  conclusions  were: 

3 . the  Army  Materiel  Systems  Analysis  Activity 

(AMSAA) /Duane  model  is  the  best  model; 
however,  the  Endiess-Burn-In  model  and 
Time-Series  Analysis  were  also  considered 
acceptable? 
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2 .  ...  that  AMSAA  model  parameters  should  be 
estimated  via  the  method  of  maximum 
-likelihood;  and 

3.  ...that  data  should  be  tracked  continuously  on 
an  individual  and  fleet  basis.  [49] 

In  1986,  Gates  (The  Analytic  Sciences  Corporation), 
analyzed  reliability  growth  data  on  nine  different  avionic 
systems  and  30  equipment  items  (line  replaceable  units) . 
They  investigated  several  reliability  growth  models  and 
selected  three  for  the  analysis  of  the  data.  The  findings 
were: 

1.  Both  the  Duane  and  AMSAA  models  were  found  to 
yield  reasonably  good  fits  to  the  data  sets. 
However,  both  were  found  to  have  limited 
utility  as  predictive  tools  because  of  the 
empirical  nature  of  the  model  parameters. 

2.  The  IBM  model  was  found  to  provide  a  more 
workable  methodology  for  growth  prediction 
because  its  parameters  lend  themselves  more 
easily  to  an  engineering  interpretation. 
[30:ES“1,  ES-2] 

Reliability  models  are  useful  for  evaluating  or 
predicting  the  reliability  potential  of  a  product.  In 
addition,  the  "quantification  provided  by  such  models  is 
most  valuable  for  proper  management  of  a  reliability 
program"  (3:11).  The  application  of  the  growth  models  will 
be  addressed  in  the  next  paragraph. 


Application  of  Growth  Models.  Gome  of  the 

reliability  growth  models  are: 

- predicting  whether  stated  reliability  objectives 

will  be  achieved; 
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- correlating  reliability  changes  with  reliability 

activities; 

- planning  for  reliability  demonstration  tests; 

- planning  for  a  reliability  improvement  warranty; 

- planning  related  to  maintenance  manpower 

utilization  and  logistics  activities; 

- performing  life  cycle  cost  analyses.  [3:11] 

Summary  of  .Traditional  Wavs  of  Assessing  Reliability. 
Reliability  prediction  is  used  to  gain  advance  knowledge  of 
the  reliability  of  the  new  systems/products.  With  the 
advance  knowledge  of  reliability,  forecasts  on  operational 
readiness,  support  costs,  spares  requirements,  and  warranty 
costs  can  be  made.  Reliability  prediction  of  highly  complex 
systems  is  based  on  the  identification  of  risks  and 
assessment  of  risks.  Testing  is  crucial  to  the  development 
of  complex  systems  in  order  to  reduce  the  risks  and  to 
identify  the  problems  early. 

Reliability  development  testing  (also  known  as 
reliability  growth  testing)  is  used  to  disclose  deficiencies 
and  to  verify  the  effectiveness  of  corx-ective  actions. 

There  are  numerous  reliability  growth  models  developed  over 
the  years.  Which  is  the  "best"  one  to  use?  The  next 
portion  of  the  literature  review  will  explore  some  practical 
questions  concerning  reliability  growth. 
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Practical  Questions  and  Answers  Associated  with  The 

BsUabilifcY  jSse 


The  objective  of  this  portion  of  the  literature  review 
was  to  relate  the  reliability  growth  concepts  to  the  theme 
of  this  study — the  reliability  gap.  The  following 
questions  and  answers  are  based  on  concepts  associated  with 
reliability  growth  planning.  These  concepts  are  from  a 
series  of  articles  written  by  Dr.  Larry  Crow  and  "have 
proven  useful  in  the  planning  and  evaluation  of  TAAF 
program"  (19:115). 

1.  Why  is  the  initial  reliability  generally  lower  than  the 
requirement  and  what  can  be  done  to  improve  it? 

2.  Are  the  requirements  realistic? 

3.  Given  that  the  requirements  are  realistic,  how  long  will 
it  take  the  initial  MTBF  to  grow  to  the  required  MTBF? 
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Figure  3.  Comparison  of  Reliability  Values  (22:387) 
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Initial  Reliability,  According  to  Dr.  Crow,  former 


Chief  of  the  Reliability  Methodology  Office  in  the 
Reliability  Division  of  the  US  Army  Materiel  Systems 
Analysis  Activity  (AMSAA) ,  the  building  blocks  for  a  new 
system  design  consist  of  four  basic  groups: 

1.  Existing  technology  used  in  the  same 
environment, 

2.  Existing  technology  used  in  a  different 
environment, 

3 «  Existing  subsystems  in  a  new  configuration,  and 

4.  New  research  and  development.  [19:115] 

The  initial  reliability  is  defined  as  "the  starting 

point  for  system-level  reliability  growth  during  development 

testing"  (19:115).  To  estimate  the  initial  reliability: 

. . .we  should  consider  all  available  information. 
This  may  include  utilizing  previous  system, 
subsystem  and  component  test  data,  historical 
experience,  engineering  expertise  and  the 
prediction  of  the  inherent  reliability.  [19:116] 

The  inherent  reliability  is  defined  as  "the  reliability 

that  is  determined  to  be  technically  achievable  for  a  basic 

system  design  within  program  time  and  cost  constraints" 

(19:115).  For  new  and  complex  designs,  the  initial 

reliability  at  the  beginning  of  development  testing  is 

generally  lower  than  the  inherent  reliability  for  numerous 

reasons,  some  of  which  may  include: 

...the  customer  use  environment  which  is  difficult 
to  design  for,  interaction  of  parts,  inaccurate  or 
incomplete  data  bases  for  predictions  and 
laboratory  test  environments  different  from  the 
use  environment.  [19:115] 
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The  growth  potential  is  defined  as  "the  maximum 
reliability  that  can  be  attained  with  the  system  design  and 
reliability  growth  management  strategy"  (22:385).  The 
elements  of  the  management  strategy  that  determine  the 
growth  potential  are  the  classification  of  Type  A  and  Type  B 
failure  modes  and  the  effectiveness  of  the  fixes  for  Type  B 
failure  modes  (22:383-304). 

Type  A  failure  modes  are  associated  with  failures  with 
no  corrective  actions.  The  management  strategy  determined 
that  it  is  not  cost  effective  to  increase  the  reliability  by 
a  design  change.  Type  B  failure  modes  are  those  that, 
if  seen  (usually  during  reliability  testing) ,  a  design 
change  will  be  attempted.  The  growth  potential  of  a  system 
will  be  attained: 

. . .when  all  Type  B  failure  modes  have  been  found 
and  a  fix  incorporated  into  the  system.  For  the 
system  design  and  management  strategy,  this  is  the 
limiting  reliability.  The  growth  potential 
reliability  may  never  actually  be  achieved  in 
practice.  [22:385] 

Dr.  Crow  further  emphasized  the  importance  of 
recognizing  that: 

. . .the  growth  potential  does  not  estimate  the 
current  reliability,  but  rather,  it  estimates  the 
maximum  reliability  that  will  be  achieved  when  all 
Type  B  failures  modes  have  been  found  and  fixed  by 
a  corrective  action.  [22:386] 

In  the  area  of  reliability  growth  analysis.  Army 
experience  has  indicated  that: 
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-..successful  programs  usually  begin  with  an 
initial  MTBF  of  at  least  1/3  or  1/4  of  the  desired 
MTBF  ([Initial  Operational  Capability]  IOC 
requirement) ,  frequently  utilize  a  fix  strategy  of 
about  95  percent  toward  failure  modes  (B  mode 
fixes)  and  generally  achieve  about  a  70  percent 
fix  effectiveness  factor.  [6:211] 


Requirement.  In  assessing  requirement,  it  is  important 
to  determine: 

. . .whether  or  not  the  requirement  is  below  the 
growth  potential  MTBF.  If  the  requirement  is  not 
below  the  growth  potential,  then  the  requirement 
cannot  be  attained  with  the  current  system  design 
and  management  strategy.  [22:385] 

Requirements  are  developed  "on  the  basis  of  user 
experience,  projected  threat,  and  system  technological 
capabilities"  (6:210) ,  Not  only  do  requirements  need  to  be 
realistic,  they  "must  also  be  affordable,  testable,  and 
achievable"  (6:210). 

Growth  Rate.  Given  that  the  requirements  are  realistic, 
affordable,  testable,  and  achievable,  the  length  of  time 
needed  to  grow  from  the  initial  MTBF  to  the  required  MTBF 
depends  on  the  growth  rate  of  the  system/equipment.  The 
growth  rate  gives  an  indication  of  how  fast  the  system 
reliability  is  improving.  The  growth  rate  will  depend  on 
"when  problems  are  detected,  fixes  incorporated  into  the 
system,  and  how  effective  the  fixes  are"  (19:117).  Morris 
and  MacDiarmid  summarized  the  growth  rate  discussion  as 
follows: 
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The  growth  rate... is  governed  by  the  amount  of 
control,  rigor,  and  efficiency  by  which  failures  are 
discovered,  analyzed,  and  corrected  through  design 
and  quality  actions.  A  large  value  of  alpha  (>0.5) 
reflects  a  hard-hitting,  aggressive  reliability 
program  with  management  support  spanning  all 
functions  of  a  knowledgeable  organization,  while  a 
low  value  of  alpha  (<.l)  reflects  the  growth  in 
reliability  that  is  due  largely  to  the  need  to 
resolve  obvious  problems  that  impact  production, 
and  to  implement  corrective  action  resulting  from 
user  experience  and  complaints.  [45:77-78] 

They  concluded  further  that  a  low  growth  rate  does  not 

necessarily  mean  a  bad  design.  In  fact,  "with  excellent 

design  and  manufacture,  the  growth  rate  could  approach 

zero"  (45:78).  In  many  growth  programs,  the  growth  rates 

were  cited  to  range  from  0.35  to  0.5  (45:78). 

Gates  identified  the  following  relationships  on  growth 

rates: 

- alpha  =  0.5,  for  a  TAF-type  test  designed  to 

stimulate  equipment  failures 

- alpha  =  0.35,  for  a  test  designed  to  simulate 

the  mission  environment 

— — alpha  =  0.2,  for  a  benign  operational  test. 
[30:7-4] 


Summary 

The  literature  review  started  with  the  definition  of 
reliability.  The  development  of  reliability  engineering 
and  understanding  the  problems  of  unr  ©liability  were  used  to 
set  the  stage  for  the  discussion  of  the  research  problem — 
the  reliability  gap.  With  the  reliability  gap  identified 
(Table  2) ,  traditional  ways  of  assessing  reliability  were 
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examined.  Specifically,  the  basic  concepts,  objectives,  and 
limitations  of  reliability  prediction,  reliability  testing, 
and  reliability  growth  were  explored.  In  addition,  some 
practical  questions  and  answers  pertaining  to  the  concepts 
of  reliability  growth  were  used  to  relate  to  the  theme  of 
this  study — the  reliability  gap.  The  next  chapter  will 
address  the  methodologies  of  how  this  study  will  be 
accomplished. 
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III.  Methodology 


Introduction 

This  chapter  describes  the  methodology  used  to 
accomplish  the  research  objectives — to  examine  the  existence 
of  the  reliability  gap,  identify  the  most  significant 
contributors,  and  explore  ways  on  how  to  minimize  the 
reliability  gap.  Data  requirement,  collection,  and  analysis 
for  the  three  phases  of  this  study  will  be  presented.  This 
chapter  concludes  with  a  summary  of  the  methodology. 


Data  Requirement 

This  portion  of  the  study  is  to  identify  the  data  that 
will  be  required  to  accomplish  the  research  objectives.  The 
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in  the  following  paragraphs. 


Phase  I.  The  data  required  to  examine  the  existence  of 
the  reliability  gap  in  tactical  airborne  radars  will  be  the 
factory  (whether  specified,  predicted,  or  demonstrated)  and 
field  MTBFs= 


The  data  required  to  identify  and  analyze  the 
major  contributors  to  the  reliability  gap  will  be  experts'/ 
practitioners'  opinions.  In  addition,  the  data  obtained 
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during  the  first  round  of  interviews  will  be  validated  by 
all  the  respondents  during  the  second  round  of  interviews. 

Phase  III.  The  data  required  to  explore  ways  on  how  to 
manage  the  reliability  gap  will  be  experts '/Practitioners ' 
opinions  from  the  interviews,  various  studies  conducted  by 
the  experts/practitioners  (DoD  &  Industry) ,  and  appropriate 
historical  studies. 

Data  Collection 

This  portion  of  the  study  is  to  describe  the 
methodologies  used  to  collect  the  data  that  were  identified 
in  data  requirement.  The  data  collection  techniques  for 
each  phase  of  the  research  are  described  below. 

Phase  I.  To  examine  the  existence  of  the  reliability 
gap  on  airborne  tactical  radars,  the  researcher  will  collect 
data  from  the  program  offices.  Air  Force  aircraft  as  well 
as  Navy  aircraft  will  be  examined. 

Phase  II.  To  identify  the  major  contributors  to  the 
reliability  gap,  the  researcher  will  conduct  two  rounds  of 
interviews.  The  first  round  of  interviews  will  be  used  to 
identify  the  experts '/Pract it ioners '  initial  opinions.  The 
second  round  of  interviews  will  be  conducted  to  validate  the 
results  obtained  from  the  first  round.  In  order  to  clarify 
what  data  will  be  collected  and  how,  the  following 
paragraphs  will  describe  the  interview  process. 
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Specifically,  the  development,  administration,  discussion, 
and  validation  of  the  interview  process. 


Interview  Questionnaire  Development.  The 
researcher  will  review  the  literature  to  develop  the 
interview  questionnaire.  The  questionnaire  will  then  be 
refined  by  the  help  of  two  different  program  office 
reliability  and  maintainability  (R&M)  engineers  and  four 
separate  Air  Force  Institute  of  Technology  (AFIT)  faculty 
members . 

The  questionnaire  will  consist  of  a  total  of  10 
questions.  In  order  to  facilitate  the  free  flow  of 
information,  most  of  the  questions  will  be  designed  to  be 
open-ended.  The  questionnaire  will  be  used  as  a  guide 
during  the  interviews  and  a  sample  of  the  questionnaire  can 
be  found  in  Appendix  B. 
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the  researcher  to  interview  as  many  reliability  experts/ 
practitioners  as  time  permits.  In  addition,  the  researcher 
intends  to  interview  as  wide  a  spectrum  of  reliability 
experts/practitioners  as  possible.  The  researcher  will  use 
the  questionnaire  to  guide  the  course  of  the  interviews. 

A  tape  recorder  will  be  used  during  the  personal  interviews. 

The  researcher  will  first  administer  the  interviews  to  a 
selective  group  of  attendees  at  the  First  Annual  Reliability 
Growth  Conference  (March  1988) .  The  selection  will  be  based 
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on  the  availability  and  willingness  of  the  attendees  to  be 
interviewed.  The  rest  of  the  interviews  will  be  conducted 
with  the  following  groups  of  individuals: 

1.  System  program  office  representatives 

2.  AFIT  professors/instructors. 

3.  Experts/Practitioners  researcher  knew  from 
previous  assignments. 

4.  References  from  interviewees. 

Interviews  Discussion.  The  researcher  selected  the 
interview  method  in  order  to  better  understand  the  existence 
of  the  reliability  gap  problem  identified  in  Phase  I  of 
this  study.  Furthermore,  interviews  will  allow  the 
researcher  to  explore  the  specific  research  problem  in 
greater  detail,  to  probe  for  additional  information  with 
follow-on  questions,  and  to  clarify  any  ambiguity. 

In  this  study,  it  is  the  intent  of  the  researcher  to 
conduct  as  many  personal  interviews  as  possible.  Due  to  the 
geographic  separation,  the  personal  interviews  will  be 
supplemented  by  telephone  interviews.  There  are  both 
advantages  and  disadvantages  involving  both  personal  and 
telephone  interviews . 

Personal  Interviews.  Emory  (26)  defined 
personal  interviewing  as  "a  two-way  conversation  initiated 
by  an  interviewer  to  obtain  information  from  a  respondent" 
(26:160).  in  order  to  be  successful,  three  conditions  must 
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be  met.  First,  the  respondents  must,  have  access  to  all 
relevant  information.  Second,  the  respondents  must 
understand  their  roles.  Finally,  the  respondents  must  be 
motivated  to  cooperate  (26;161). 

In  this  study,  the  requirements  identified  by  Emory  will 
be  met  at  the  beginning  of  each  interview  session.  The 
researcher  will  first  explain  the  purpose  of  the  study  and 
why  the  respondents'  participation  is  so  important.  The 
researcher  will  then  explain  to  the  respondents  how  they  can 
change  their  answers  during  the  second  round  of  interviews. 
In  addition,  the  researcher  will  explain  to  the  respondents 
that  the  use  of  the  tape  recorder  is  to  capture  the 
enormous  flow  of  information  during  the  interview.  The 
recorder  is  not  used  to  quote  respondents  out  of  context. 

Telephone  Interviews.  Emory  identified  that 
the  telephone  interview  possesses  some  of  the  same 
characteristics  as  the  personal  interview.  The  main 
advantage  of  telephone  interviews  is  their  low  cost. 

Another  advantage  is  that  "interviewer  bias  is  reduced  by 
using  telephones"  (26;170). 

Interview  Validation.  In  this  study,  the  results 
from  the  first  round  of  interviews  will  be  validated  by  a 
second  round  of  interviews.  The  purpose  of  the  second  round 
is  to  validate  the  results  obtained  irom  the  first  round  and 
to  give  the  respondents  a  chance  to  provide  additional 
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comment(s)  to  the  summarized  results  from  the  first  round. 
The  following  paragraphs  will  describe  how  the  validation 
process  of  the  interviews  will  be  conducted  in  this  study. 

The  researcher  will  conduct  the  interviews  and  summarize 
the  results  from  the  first  round.  The  summarized  results, 
along  with  the  respondents'  .initial  answers,  will  be  sent 
back  to  the  participants  for  additional  comment (s) .  Data 
will  be  gathered  during  both  rounds. 

Round  One.  The  researcher  will  first  define 
the  reliability  gap  and  ask  the  interviewees  to  identify  the 
major  contributors  to  the  gap.  After  the  respondents  have 
named  the  major  contributors,  the  researcher  will  ask  them 
to  rank  order  the  contributors  they  have  identified. 

Round  Two.  During  the  second  round  of 
interviews,  the  respondents  will  be  asked  to  validate  their 
initial  responses  and  to  provide  additional  comment (s)  to 
the  summarized  results  from  the  other  respondents. 

Phase  III.  To  explore  ways  on  how  to  minimize  the 
reliability  gap,  the  researcher  will  collect  the  experts'/ 
practitioners'  opinions  from  the  same  questionnaire  used 
during  Phase  II  of  this  study.  The  additional  data  sources 
will  be  various  studies  conducted  by  til  i  experts/ 
practitioners  and  appropriate  historical  findings. 
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This  portion  of  the  study  describes  how  the  collected 
data  will  be  analyzed.  The  data  analysis  for  each  phase  of 
this  study  will  be  presented  in  the  following  paragraphs. 

Phase  I.  To  determine  the  existence  of  the  reliability 
gap  in  airborne  tactical  radars,  the  researcher  will  examine 
the  radars  on  Air  Force  as  well  as  Navy  aircraft. 

rhase  II.  To  identify  and  analyze  the  major 
contributors  to  the  reliability  gap,  the  researcher  will  use 
personal  and  telephone  interviews  to  collect  the  initial 
responses.  In  addition,  the  researcher  will  conduct  a 
second  round  of  interviews  to  validate  the  results  obtained 
from  the  first  round. 

In  order  to  present  a  detailed  description  of  how  the 
researcher  intends  to  analyze  the  results  using  two  rounds 
of  interviews,  the  following  paragraphs  will  describe  in 
detail  the  objectives  and  the  specific  methodologies  used 
for  recording,  reducing,  ranking,  and  reorganizing  the 
collected  data. 

Recording  the  Data.  The  objective  is  to  record 
the  number  of  times  a  particular  contributor  will  be 
identified  by  the  interviewees.  During  the  interviews,  the 
respondents  will  be  asked  to  identify  the  major  contributors 
to  the  reliability  gap.  The  total  number  of  times  a 
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particular  contributor  is  mentioned  by  the  respondents  will 
be  recorded. 

Reducing  the  Data.  The  objective  is  to  examine 
the  most  significant  contributors.  From  the  interviews,  the 
total  number  of  major  contributors  will  be  identified. 

Using  the  number  of  responses  as  the  determining  factor,  the 
major  contributors  will  be  divided  into  significant  and 
insignificant  contributors,  significant  contributors  will 
be  defined  as  contributors  with  a  significant  number 
(visually  determined)  of  responses.  The  number  of  responses 
will  be  classified  as  significant  if  they  are  closely 
grouped  together  toward  the  high  end  of  the  scale.  If  there 
is  an  obvious  break  in  the  number  of  responses,  the  rest  of 
the  contributors  will  be  classified  as  insignificant. 

Ranking  the  Data.  The  objective  is  to  present  the 
rank  order  averages  of  the  most  significant  contributors. 
During  the  interviews,  the  respondents  will  be  asked  to  rank 
order  the  major  contributors.  From  the  responses,  the 
researcher  will  assign  a  "10"  to  the  number  one  contributor, 
a  "9"  to  the  number  two  contributor,  etc.  Consequently,  for 
any  particular  contributor,  a  group  of  numbers  will  be 
assigned.  To  analyze  the  relative  importance  of  the  major 
contributors,  the  averages  of  the  rank  order  of  the 
contributors  will  be  calculated. 
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Reorganizing  the  Data.  The  objective  is  to 
group  the  data  into  different  categories  in  order  to  enhance 
data  analysis.  Basically,  all  the  interviewees  can  be 
categorized  into  four  different  organizations:  Air  Force, 
Army,  Navy,  and  Industry.  Using  the  same  rank-order 
procedure  described  above,  the  researcher  will  classify 
the  responses  into  subcategories  (All  vs  DoD  vs  Industry) 
and  compare  the  results. 

Phase  III.  In  order  to  explore  ways  on  how  to  minimize 

the  reliability  gap,  the  researcher  will  use  the  data  that 
* 

will  be  collected  during  the  interviews.  In  addition,  the 
researcher  will  also  analyze  various  studies  conducted  by 
the  interviewees  and  appropriate  historical  studies. 

gummary 

Various  techniques  will  be  used  in  this  study  to 
investigate  the  research  problem — the  reliability  gap.  Table 
3  provides  a  summary  of  the  methodology  which  will  be 
used  in  this  research.  The  next  chapter  presents  the 
findings  obtained  in  this  research. 
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Table  3 


Methodology  Summary 


Phase  Objective 


Data  Requirement 
Data  Collection 
Data  Analysis 


I  To  examine  the 
existence  of  the 
reliability  gap 

II  To  identify  the  major 
contributors  to 
the  reliability  gap 


III  To  explore  ways 

on  how  to  minimize 
the  reliability  gap 

-Practical  guidelines 
on  reliability 
growth 


-Tactical  airborne  radars 
on  Air  Force  and  Navy 
aircraft 

-Experts '/Practitioners ' 
opinions 

-Interviews  conducted 
Interviews  validated 

- Personal 

- Telephone 

-Major  contributors  analyzed 

- Most  significant 

contributors  identified 
and  rank  ordered 

-Reliability  management  data 

-Experts'  opinions,  various 
program  offices'  data, 
current  studies  conducted  by 
the  interviewees,  and 
appropriate  historical 
studies 

-Analyze  the  collected  data 
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IV.  Findings 


Introduction 

This  chapter  presents  the  findings  used  to  answer  the 
research  questions — the  existence  of  the  reliability  gap 
in  airborne  tactical  radars,  the  relative  importance  of  the 
major  contributors,  and  practical  approaches  on  how  to 
minimize  the  gap.  The  findings  for  the  three  phases  of 
this  study  will  be  presented.  This  chapter  concludes  with  a 
summary  of  the  findings. 

Phase  I  Findings 

Both  Air  Force  and  Navy  aircraft  were  examined.  The 
data  obtained  from  NAVAIR  is  used  in  this  study  to  present 
the  existence  of  the  reliability  gap  in  tactical  airborne 
radars.  Figure  4  is  a  courtesy  slide  obtained  from  NAVAIR. 
It  is  interesting  to  note  that  when  the  field  MTBF  was 
better  than  the  specified  MTBF,  the  gap  seemed  to  be  very 
small.  But  when  the  field  MTBF  was  less  than  the  specified 
MTBF,  the  gap  seemed  to  be  significantly  large.  With  the 
existence  of  the  reliability  gap,  it  becomes  obvious  to  ask 
the  next  set  of  questions.  What  causes  the  gap?  What  are 
the  major  contributors?  What  can  be  done  to  minimize  the 
gap  and  hence  maximize  operational  readiness  and  optimize 
life  cycle  costs? 
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Phase  II  Findings 


Personal  and  telephone  interviews  were  conducted.  In 
addition,  a  second  round  of  interviews  was  performed  to 
validate  the  findings  from  the  first  round.  Experts'/ 
Practitioners'  opinions  on  the  major  contributors  to  the 
reliability  gap  were  then  consolidated. 

Both  personal  and  telephone  interviews  were  conducted. 

In  this  study,  personal  interviews  constituted  75%  of  all 
the  interviews.  The  length  of  the  personal  interviews 
ranged  from  30  to  180  minutes,  depending  on  how  the 
individual  interviewee  responded  to  the  open-ended 
questions.  Most  of  the  personal  interviews  lasted  for  90 
minutes.  The  length  of  the  telephone  interviews  ranged  from 
10  to  45  minutes.  In  this  study,  most  of  the  telephone 
interviews  were  accompanied  by  mail  responses. 

A  total  of  40  experts/practitioners  were  interviewed. 
Thirteen  were  participants  (mostly  Industry)  the  researcher 
met  during  the  First  Reliability  Growth  Conference  held  in 
Boston,  March  1988.  Seven  were  participants  from  various 
program  offices  and  another  seven  were  instructors  from 
AFIT.  Nine  were  participants  recommended  by  the  other 
respondents  and  the  remaining  four  were  experts/ 
practitioners  the  researcher  knew  from  previous  assignments. 
The  data  collected  from  the  interviews  were  analyzed 
according  to  the  methodologies  described  in  Chapter  III. 
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The  following  paragraphs  present  the  findings  from  the 
interviews.  The  findings  were  grouped  into  the  following 
categories: 


1.  Round  One  of  the  First  28  Interviewees. 

2.  Round  Two  of  the  First  28  Interviewees. 

3.  Round  One  of  All  38  Interviewees. 

4.  Round  Two  of  All  38  Interviewees. 


1 .  Round  one  of  the  First  28  Interviewees.  The, 
researcher  intervi jwed  a  total  of  30  participants.  Two 
of  the  participants  were  excluded  from  the  summarized 
results  because  of  their  special  expertise.  Their 
responses  were  highly  concentrated  in  one  specific  area. 

The  researcher  interviewed  those  people  in  order  to  use 
their  expertise  in  later  parts  of  this  study. 

In  the  interest  of  time,  the  researcher  summarized  the 
results  from  the  first  group  of  28  interviewees.  The 
researcher  then  sent  the  summarized  results,  along  with  the 
respondents'  initial  answers,  back  to  the  participants  for 
feedback.  The  specific  analysis  of  the  collected  data  of 
this  part  of  the  research  was  described  in  Chapter  III. 
Detailed  results  are  attached  as  Appendices  E,  F,  and  G. 

2.  Round  Two  of  First  28  Interviewees.  During  the 
second  round  of  interviews,  the  participants  had  the 
opportunity  to  modify  their  initial  answers.  In  addition, 
the  researcher  had  the  opportunity  to  verify  and  clarify  the 
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answers.  Most  of  the  participants  were  contacted  by 
telephone  during  this  round  of  interviews  and  the 
researcher  was  able  to  get  responses  from  all  the 
participants.  Of  the  28  participants,  only  eight  changed 
the  rank  order  of  the  major  contributors.  Detailed  results 
are  attached  as  Appendices  H,  I,  and  J. 

3 .  Round  One  of  All  38  Interviewees .  The 
additional  10  participants  were  contacts  the  researcher 
made  during  the  early  part  of  this  research.  Due  to  the 
different  schedules  of  both  the  participants  and  the 
researcher,  it  took  some  time  to  conduct  the  additional 
interviews . 

The  additional  10  interviews  were  conducted  in  the  same 
manner  as  the  first  28.  The  participants  were  first  asked 
to  respond  to  the  interview  questionnaire.  After  the 
researcher  received  the  initial  responses  from  the 
participants,  the  summarized  results  (obtained  from  the 
first  28  participants) ,  along  with  their  initial  answers, 
were  sent  back  to  the  participants  for  feedback.  For  the 
overall  analysis,  the  researcher  combined  the  additional 
data  from  the  remaining  10  interviewees  with  the  summarized 
results  obtained  from  the  first  28  interviewees.  The 
specific  details  for  the  analysis  of  the  collected  data  were 
described  in  Chapter  III.  Detailed  results  are  attached  as 
Appendices  K,  L,  and  M. 
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4.  Round  Two  of  All  38  Interviewees.  The  second 
round  of  interviews  for  this  part  was  conducted  in  the  same 
manner  as  round  two  for  the  first  28  interviewees.  During 
this  round  of  interviews,  both  the  participants  and  the 
researcher  were  able  to  provide  feedback  on  the  initial 
responses.  The  participants  were  able  to  modify  their 
initial  answers  if  they  so  desired  or  just  clarify  their 
meanings  to  their  initial  answers.  The  researcher  was  able 
to  verify  and  clarify  the  responses  and  thereby  minimize 
misinterpretation  on  some  of  the  initial  answers.  Detailed 
results  are  attached  as  Appendices  N,  0,  and  P. 

Discussion  on  the  Collected  Data.  From  the 
collected  data,  the  contributors  were  recorded,  reduced, 
ranked,  and  reorganized  according  to  the  methodologies 
described  in  Chapter  III.  The  following  paragraphs 
describe  the  specific  findings  of  the  collected  data. 

Recording  the  Data.  From  the  interviews,  the 
major  contributors  to  the  reliability  gap  were  identified 
and  rank  ordered.  The  total  number  of  times  a  particular 
contributor  was  mentioned  by  the  respondents  was  recorded. 
Figure  6  (Round  One)  and  Figure  7  (Round  Two)  present  the 
summaries  of  the  number  of  responses  of  the  contributor 
identified  by  28  interviewees.  Figure  8  (Round  One)  and 
Figure  9  (Round  Two)  present  the  summaries  of  the  number  of 
responses  of  the  contributor  identified  by  38  interviewees. 
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28  interviewees 


Figure  6.  Round  Two  of  28  Interviewees 


38  Interviewees 

(28  DoD/10  Industry) 


Figure  7.  Round  One  of  38  Interviewees 


Figure  8.  Round  Two  of  38  Interviewees 


Reducing  the  Data.  From  the  interviews,  12 


major  contributors  were  identified.  By  examining  Figures  6 
through  9,  the  major  contributors  can  be  grouped  into 
significant  and  insignificant  contributors.  The  significant 
contributors  had  a  fairly  high  number  of  responses  and  the 
insignificant  contributors  had  a  fairly  low  number  of 
responses.  Using  the  number  of  responses  as  the  determining 
factor,  the  significant  contributors  from  round  one  of  the 
first  28  interviewees  were  determined  to  be:  Environment, 
Data,  Prediction,  Manufacturing,  and  Design.  However,  by 
examining  Figures  6  and  9,  the  most  significant  contributors 
were  determined  to  be:  Data,  Prediction,  Environment, 
Manufacturing,  Design,  and  Management.  An  interesting 
finding  in  this  study  was  the  difference  in  the  results 
between  the  sample  sizes  (28  vs  38) .  This  Interesting 
phenomenon  is  described  in  the  following  pazagraphs.  The 
difference  in  the  results  between  the  sample  sizes  was  not 
part  of  the  original  intent  of  this  study;  however,  it  is 
interesting  to  note  the  existing  differences.  The  most 
significant  differences  were  in  the  areas  of  Management  and 
Data. 

With  28  interviewees,  management  was  net  determined 
(using  the  procedures  described  in  Chapter  ill \  as  one  of 
the  most  significant  contributors;  however,  with  38 
interviewees,  management  was  definitely  considered  as  one 
of  the  most  significant  contributors.  Ey  examining  the 
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rank-order  averages  of  management  during  round  two 
interviews  with  both  28  and  38  interviewees,  it  is 
interesting  to  note  that  management  has  the  highest  averages 
in  both  casts. 

Another  interesting  contributor  was  Data.  By  strictly 
using  the  interviewees  from  Industry  (five  interviewees)  and 
using  the  procedures  identified  in  Chapter  III,  Data  would 
not  have  been  considered  as  one  of  the  most  significant 
contributors.  However,  the  rank-order  averages  of  data  from 
Industry  we-e  as  high  as  those  from  DoD,  if  not  higher. 

Ranking  the  Data.  After  the  identification  of 
the  most  significant  contributors,  the  rank-order  averages 
of  those  contributors  were  calculated.  The  specific 
methodology  was  described  in  Chapter  III.  The  rank-order 
averages  were  used  to  analyze  the  relative  importance  of  the 
most  significant  contributors. 

The  most  significant  contributors,  along  with  their 
rank-order  averages,  from  both  rounds  of  interviews  of  28 
interviewees  were  determined  to  be:  Environment,  Data 
Prediction,  Manufacturing,  and  Design  (see  Figures  10  and 
11) .  The  most  significant  contributors  from  both  rounds  of 
interviews  of  38  interviewees  were  determined  to  be: 
Environment,  Management,  Data,  Prediction,  Manufacturing, 
and  Design  (see  Figures  12  and  13) .  To  investigate  the 
relative  importance  of  management  among  28  interviewees,  the 
researcher  included  Figures  14  and  15. 
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Figure  9.  Most  Significant  Contributors  (Round  One:  28) 


28  interviewees 


38  Interviewees 

(28  DoD/10  Industry) 


Figure  12.  Most  Significant  Contributors  (Round  Two:  38) 


28  Interviewees 

(23  DoD/5  Industry) 


Figure  13.  Most  Significant  Contributors  (Round  One:  28) 


28  Interviewees 

(23  DoD/5  Industry) 


Figure  14.  Most  Significant  Contributors  (Round  Two:  38) 


Reorganizing  the  Data.  In  order  to 
graphically  compare  the  relative  importance  of  the  most 
significant  contributors,  the  researcher  used  the  average 
rank  orders  of  the  contributors  to  reorganize  the  data  on  a 
linear  scale  (0  to  10) .  Due  to  the  subjective  nature  of 
this  study,  the  exact  location  of  any  contributor  on  the 
linear  scale  was  not  as  important  as  its  relative  location. 
For  example,  when  the  contributors  are  closely  grouped 
together,  it  is  very  difficult  to  distinguish  which  one  is 
more  important  than  the  other.  However,  when  there  is  a 
break  between  the  contributors,  it  often  indicates  that  one 
contributor  is  definitely  more  important  than  the  other. 

By  looking  at  the  data,  the  rank  orders  of  any 
contributor  did  not  seem  to  change  much  from  one  round  of 
interviews  to  another.  For  example,  the  top-ranking 
contributors  during  the  first  round  of  interviews  were  still 
the  top  ranking  contributors  after  the  second  round  of 
interviews. 

It  is  not  surprising  to  note  that  the  DoD  rankings  match 
closely  with  the  Overall  (DoD  &  Industry)  rankings 
since  DoD  interviewees  constituted  75%  of  the  total 
interviewees.  However,  it  is  interesting  to  note  that 
between  the  DoD  and  Industry  rankings,  the  most  significant 
difference  was  in  the  average  rank  order  for  Management  and 
the  number  of  responses  for  Data.  Management  was  ranked 
very  high  among  DoD  personnel  and  not  so  high  with  Industry 
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personnel.  In  the  area  of  Data,  both  the  DoD  and  Industry 
communities  ranked  it  fairly  high;  however,  the  number  of 
responses  from  the  two  communities  differed  drastically. 

For  example,  24  of  the  28  (86%)  DoD  respondents  mentioned 
Data  as  one  of  the  major  contributors,  whereas  only  3  of  the 
10  (30%)  Industry  respondents  mentioned  Data. 

The  linear  scale  ranges  from  0  to  10  are  used  to 
display  the  most  significant  contributors.  Because  the 
rank-order  averages  were  concentrated  toward  the  top  half  of 
the  scale,  only  the  numbers  from  7  to  10  were  labeled  on  the 
tables.  Table  4  presents  the  relative  importance  of  the 
most  significant  contributors  to  the  reliability  gap  from 
the  results  of  the  first  28  interviewees.  The  first  column 
represents  the  responses  from  both  DoD  and  Industry 
participants.  The  second  column  represents  the  responses 
from  DoD  participants  only,  and  the  third  column  represents 
the  responses  from  Industry  participants  only.  The  number 
in  parentheses  after  each  contributor  represents  the  number 
of  interviewees  responding  to  that  particular  contributor. 
Table  5  presents  the  relative  importance  of  the  most 
significant  contributors  to  the  reliability  gap  from  the 
results  from  the  second  round  of  interviews  of  the  first  28 
interviewees.  Tables  6  and  7  present  the  relative 
importance  of  the  most  significant  contributors  to  the 
reliability  gap  from  the  results  from  both  rounds  of 
interviews  of  38  interviewees  (see  Tables  4  through  7) . 
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Table  4 .  Relative  Importance  of  the.  Most 
Significant  Contributors  to  the 
Reliability  Gap  (Round  One:  28) 


Rank-  DoD  &  Industry  DoD  Industry 

Order  (28)  (23)  (5) 

Average 


10 


Mgmt (6) 


Mgmt ( 5 ) 


Mfg<3) 


9 


8 


Envmt(18) 
Data (18) 


Freds ( 16) 
Mfg(15) 


Design (12) 


Envmt(15) 
Data (17) 

Preds(14) 

Mfg ( 12 ) 
Design(9) 


Design ( 3 ) 
Data(l) 

Preds (2) 

Mgmt ( 1 ) 
Envmt ( 3 ) 


7 
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Table  5.  Relative  Importance  of  the 
Significant  Contributors  to 
Reliability  Gap  (Round  Two: 
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9. 

Envmt ( 15) 

Desigri(3) 
Data ( 1) 

Data  ^ 18 ) 

Envmt (18) 

Data (17) 
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8. 
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Preds ( 17 ) 

Mfg (15) 

Design (12) 

Preds (15) 

Mfg (12) 

Design(9) 

Mgmt ( 1 ) 

Emrmt  ( 3 ) 
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Table  6.  Relative  Importance  of  the  Most 
Significant  Contributors  to  the 
Relability  Gap  (Round  One:  38) 


Bankr  POD  &  industry  Don 
Order  (38)  (28) 
Average 


10 


9 


8 


Envmt (18) 
Mgmt ( 8 ) 


Envmt (24) 
Mgmt (12) 
Data (27) 

Preds (25) 

Mfg (21) 
Design (13) 


Data (24) 
Preds ( 18 ) 

Design (10) 
Mfg ( 15) 


Industry 

(10) 


Data (3) 
Mfg (6) 

Design (3 ) 

Preds (7) 

Mgmt ( 4 ) 
Envmt ( 6 ) 
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Table  7. 


Relative  Importance  of  the  Most 
Significant  Contributors  to  the 
Reliability  Gap  (Round  Two:  38) 


Industry 
(10) 

Average 


10 


Rank-  DoD  &  Industry  i,)pQ 

Order  (38)  (28) 
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Mgmt  ( 14) 


Design (3) 
Data (3) 
Mfg (6) 

Mgmt (9) 

Envmt (18) 
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Data (27) 

Envmt (24)  Data (24)  Preds(7) 

Preds (26)  Preds(19) 

Mfg  (21) 

Design (14)  Design(ll) 

Mfg (15)  Mgmt (4) 

Envmt ( 6 ) 
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Discussion  of  the  Findings.  From  the  first  round  of 
interviews,  a  second  round  was  conducted  to  validate  the 
findings.  For  the  purpose  of  trying  to  understand  the 
makeup  of  the  different  contributors,  the  following 
paragraphs  provide  a  descriptive  summary  of  the  comments 
cited  by  the  interviewees  (also  see  Appendix  C,  a  summary  of 
che  findings  of  the  first  28  interviewees) .  In  addition, 
various  studies  (conducted  by  the  interviewees)  and  possible 
solutions  (suggested  by  the  interviewees)  were  also  included 
in  the  discussion.  The  contributors  are  presented  in  the 
order  shown  in  Table  7,  using  the  DoD  &  Industry  column. 
Since  all  the  contributors  identified  are  significant,  the 
contributors  with  the  highest  number  of  responses  will  be 
described  first. 

Data.  Twenty-seven  out  of  38  interviewees 
(71%)  identified  data  as  one  of  the  major  contributors  to 
the  reliability  gap.  Most  experts  felt  that  the  problems 
with  data  were  in  the  areas  of  data  definition  and  data 
collection. 

Under  the  topic  of  data  definition,  most  respondents 
mentioned  the  difficulty  associated  with  the  classification 
of  failures  and  scoring  procedures.  The  difficulties 
include  the  determination  of  relevant  vs  nonrelevant 
failures,  the  definition  of  a  failure  vs  a  critical 
failure,  the  contractual  vs  operational  failures. 
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inherent  vs  induced  failure,  Government  Furnished  Equipment 
(GFE)  vs  non-GFE  failures,  and  hardware  vs  software 
problems.  The  difficulty  also  includes  the  definition  of 
time,  such  as  flying  vs  operating  hours. 

Under  the  topic  of  data  collection,  most  of  the 
interviewees  considered  the  current  DoD  data  collection 
system  as  inaccurate,  untimely,  and  incomplete.  They  also 
mentioned  that  there  is  a  lack  of  motivation  from  the  data 
collectors.  The  lack  of  motivation  is  mainly  due  to  the 
lack  of  positive  feedback  and  an  overwhelming  amount  of 
negative  feedback.  In  addition,  most  of  the  respondents 
felt  that  the  current  data  collection  system  is  used  as  a 
manhour  accounting  system,  to  ensure  the  documentation  of  a 
full-day’s  work,  rather  than  as  a  system  to  collect  accurate 
reliability  data. 

It  is  also  interesting  to  note  the  comments  from 
Dr.  Ben  Williams  (Director,  Center  of  Excellence  for  R&M) , 
who  has  over  25  years  of  experience  in  reliability  and 
maintainability  management.  According  to  him,  the  problem 
is  that  "...we  do  not  know  how  to  analyze  the  collected 
data."  He  thinks  the  current  data  system  is  not  perfect  but 
is  adequate  to  provide  the  necessary  information — that  is, 
the  capability  to  identify  where  the  problem  areas  are.  He 
further  explained  that  it  is  not  economically  feasible,  nor 
it  is  necessary,  to  investigate  every  failure,  provided  that 
the  failure  is  not  safety-related  and  does  not  affect  the 
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operational  availability  of  the  weapon  system.  What  is 
necessary  is  to  have  the  right  people  (preferably  R&M 
engineers)  to  analyte  the  da.  a  and  identify  the  failures 
that  warrant  the  investigation.  That  is,  the  benefits  from 
corrective  action (s)  outweigh  the  resources  expended  for  the 
investigation . 


Data:  Current  Studies.  An  R&M  Data 

Deficiencies  Tiger  Team  was  formed  in  February  1987  to 
investigate  the  Maintenance  Data  Collection  (MDC)  system  and 
its  relationship  with  R&M  requirements  (Appendix  Q) .  The 
team  collectively  generated  more  than  60  recommendations  which 
were  then  consolidated  into  15  items.  Due  to  the  relative 
importance  of  data  as  compared  to  the  other  contributors  on 
the  linear  scale,  this  study  included  the  problem  descriptions 
of  the  15  items  as  Appendix  Q. 

Data:  Possible  Solution.  Many  solutions 

were  suggested.  From  this  study,  the  objective  of  the 
data  definition  problem  is  to  come  up  with  a  set  of  standard 
R&M  measurable  terms.  The  objective  of  the  data  collection 
problem  is  to  come  up  with  a  common  data  base.  With  the 
standard  set,  the  tool  for  the  implementation  of  the 
suggested  concepts  is  a  joint  data  classification  board 
meeting  to  clarify  the  specific  classification  and 
definition.  In  addition,  the  joint  data  classification 
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board  can  generate  a  set  of  data  which  can  be  used  as  the 
common  data  base. 

On  the  more  sophisticated  side  of  data  collection, 
Colonel  Abrams,  former  F-15  Deputy  Program  Manager  for 
Logistics  { DPML) ,  suggested  the  use  of  an  artificial 
intelligence  (AI) -based  system.  With  an  AI -based  system, 
real-time  data  under  operational  conditions  can  be 
accurately  captured. 

In  the  meantime,  is  there  a  possible  solution  to  the 
data  collection  deficiency?  It  has  been  suggested  that  the 
current  data  collection  system  is  not  perfect,  and  it  never 
will  be.  It  is,  however,  adequate  to  provide  the  necessary 
information.  The  deficiency  is  not  the  collected  data,  per 
se.  The  deficiency  is  in  not  knowing  how  to  analyze  the 
collected  data.  A  possible  solution  is  to  have  the  "right" 
people  (preferably  R&M  engineers)  analyze  the  data,  identify 
the  magnitude  of  the  problem,  and  conduct  a  special  study  on 
the  problem  if  the  problem  has  been  identified  as 
economically  and  operationally  beneficial. 

Prediction.  The  word  "prediction,"  as  used  in  this 
phase  of  the  study,  refers  to  the  prediction  techniques. 
Twenty-six  out  of  38  interviewees  (68%)  identified 
prediction  techniques  as  one  of  the  major  contributors. 

About  half  the  experts  attributed  the  problem  to  the 
techniques  used  in  MIL-HDBK-217,  The  assumptions  on  the 
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prediction  techniques  in  MIL-HDBK-217  were  cited  by  the 
experts  to  be  questionable.  In  addition,  the  predictors  in 
the  handbook  were  considered  to  be  operationally 
nonrepresentative.  The  predictors  do  not  consider 
interconnections,  and  there  are  no  good  predictors  for 
hydromechanical  failure  rates.  Other  problems  with  the 
prediction  technique  were  stated  to  be  associated  with  the 
improper  use  of  reliability  models.  In  addition,  most  of 
the  experts/practitioners  suggested  that  the  prediction 
techniques  accounted  for  the  bulk  of  the  reliability  gap. 

As  far  as  MIL-HDBK-217  is  concerned,  both  Dr.  Crow 
(18-23)  and  Mr.  Gibson  (30)  stated  that  its  intended  purpose 
is  to  serve  as  a  trade-off  tool  for  early  design  and  not  to 
predict  the  performance  of  mature  systems.  According  to 
them,  people  using  MIL-HDBK-217  to  make  predictions  must 
first  understand  its  intended  purpose  and  use  it 
accordingly. 

Prediction:  Possible  Solution.  Through  years 

of  experience,  Mr.  Trakas  (NAVAIR:  Head,  R&M  Branch) 
developed  a  practical  approach  of  relating  the  reliability 
parameters.  The  objective  of  this  approach  is  to  "provide  a 
repeatable,  logical  approach  toward  establishing  realistic 
reliability  requirements"  (66:103)  for  the  acquisition 
process.  Appendix  R  is  taken  from  the  article  written  by 
Mr.  Trakas  and  it  outlines  this  practical  approach.  It 
starts  with: 
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...the  operational  requirement  defining  the  minimum 
acceptable  value  of  reliability,  in  operational 
terms,  consistent  with  meeting  tt  a  program 
objectives.  The  operational  requirement  is  then 
translated  into  the  DCP/TEMP  (Decision  Coordinating 
Paper/Test  and  Evaluation  Master  Plan)  threshold 
MFHBF  (Mean  Flight  Hours  Between  Failure)  taking 
into  account  the  logistics  inputs  of  MFKBMA  (Mean 
Flight  Hours  Between  Maintenance  Actions) , 
cannibilizations,  on-aircraft  repair,  MFHBR  (Mean 
Flight  Hours  Between  Removal)  if  applicable,  and 
the  false  removal  rate....  This  reliability  is  in 
operational  terms  and  must  be  translated  into  an 
MTBF  (Mean  Time  Between  Failure)  requirement  for 
the  contractor.  In  order  tr>  accomplish  this 
translation,  an  S/F  (System/Flight)  Ratio  must 
first  be  applied.  The  S/F  ratio  takes  into  account 
the  fact  that  this  type  of  equipment  is  on  and 
operating  more  than  it  is  flying  (assuming  a  100% 
duty  factor  in  the  aircraft)  because  of  Pre-Flight, 
Flight,  Post-Flight,  and  maintenance  activities. 
This  ratio  then  yields  an  MTBF  threshold. ...  an 
experience  factor  of  25%,  based  on  the  Duane 
philosophy,  is  applied  yielding  a  Lower  Test 
MTBF. —  In  accordance  with  MIL-STD-781C,  assuming 
a  typical  test  using  Test  Plan  IIIC,  there  is  a  2:1 
discrimination  ratio.  This  yields  an  Upper  Test 
MTBF.  [66:105] 

The  safety  margin  percentage  is  used  as: 

...the  rule  of  thumb  that  there  is  a  good  chance 
that  the  specified  requirement  can  be  achieved  and 
demonstrated  if  the  predicted  reliability  is  25 
percent  greater  than  the  Upper  Test  MTBF.  [66:106] 

According  to  Mr.  Trakas,  the  methodology  described  is 
for  avionics.  Similar  methodologies  can  be  developed  for 
different  systems.  In  fact,  the  same  type  of  methodology 
was  developed  and  documented  in  Air  Force  Logistics  Coiamand 
Pamphlet  ( AFLCP)  800-3,  April  1973. 


Environment.  Twenty-four  out  of  38  interviewees 
(63%)  identified  environment  as  one  of  the  major 
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contributors.  Most  of  the  participants  responded  that  most 
of  the  problems  associated  with  environment  are  attributed 
to  the  misapplication  and  misinterpretation  of  the  * 
operational  environment — the  lack  of  good  translational 
factor  between  operational  and  laboratory  environment.  The 
operational  environment  is  harsh  and  difficult  to  predict. 

A  few  of  the  respondents  think  that  we  do  not  really 
understand  the  operational  environment,  and  we  do  not 
accomplish  adequate  testing,  to  find  out  about  the 
operational  environment.  In  addition,  the  predicted 
reliability  is  mostly  at  the  parts  or  subsystems  levels. 
System  integration  and  interfacing  is  seldom  considered.  No 
specific  solution  was  cited. 

Manufacturing .  Twenty-one  of  the  38  interviewees 
(55%)  identified  manufacturing  as  one  of  the  major 
contributors  to  the  reliability  gap.  Most  of  the 
respondents  felt  that  the  problem  with  the  manufacturing 
processes  is  in  the  areas  of  producibility  and  quality. 

They  felt  that  a  robust  design  can  absorb  a  lot  of  the 
producibility  problems.  As  far  as  quality  is  concerned, 
most  of  the  respondents  felt  that  it  is  also  contractor- 
dependent,  on  whether  the  particular  contractor  believes  in 
a  total  quality  management  program. 

Manufacturing:  Possible  Solution.  Several 
other  respondents  made  the  following  suggestions: 
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1.  Encourage  the  design  engineers  to  work  closely  with 
the  manufacturing  engineers  by  collocating  the  two 
work  areas. 

2.  Maximize  the  use  of  standardized  parts.  With  the 
use  of  standardized  parts,  the  risks  to  parts 
performance,  costs,  etc.,  can  be  reduced. 

3.  Minimize  parts  count.  The  reasoning  behind  this 
suggestion  is  with  fewer  parts,  there  are  fewer 
failures  associated  with  the  parts  or  interfaces. 

In  addition,  Major  Farr  (AFIT  Professor:  Contracting 
Management) ,  who  has  more  tnan  eight  years  of  experience  in 
manufacturing,  advocated  the  use  of  the  Willoughby 
templates  in  DoD  Directive  4245.7.  Willis  J.  Willoughby, 
Jr.,  is  the  Chairman,  Defense  Science  Board  Task  Force, 
Transition  from  Development  to  Production.  In  the  area  of 
quality,  several  respondents  mentioned  Taguchi's  quality 
philosophy  to  design.  Genichi  Taguchi  "is  frequently 
mentioned  along  with  w.  Edwards  Deming,  Kaoru  Ishikawa,  and 
J.  M.  Juran"  (36:21).  The  following  seven  points  explain 
the  basic  elements  of  Taguchi's  quality  philosophy: 

1.  An  important  dimension  of  the  quality  of  a 
manufactured  product  is  the  total  loss 
generated  by  that  product  to  society. 

2.  In  a  competitive  economy,  continuous  quality 
improvement  and  cost  reduction  are  necessary 
for  staying  in  business. 

3.  A  continuous  quality  improvement  program 
includes  incessant  reduction  in  the  variation 
of  product  performance  characteristics  about 
their  target  values. 

4.  The  customer's  loss  due  to  a  product's 
performance  variation  is  often  approximately 
proportional  to  the  square  of  the  deviation 
of  the  performance  characteristic  from  its 
target  value. 
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5.  The  final  quality  and  cost  of  a  manufactured 
product  are  determined  to  a  large  extent  by  the 
engineering  designs  of  the  product  and  its 
manufacturing  process. 

6.  A  product's  (or  process')  performance 
variation  can  be  reduced  by  exploiting  the 
nonlinear  effects  of  the  product  (or  process) 
parameters  on  the  performance  characteristics. 

7 .  Statistically  planned  experiments  can  be  used 
to  identify  the  settings  of  product  (and 
process)  parameters  that  reduce  performance 
variation.  [36:21] 

Design.  Fourteen  of  the  38  interviewees  (37%) 
identified  design  as  one  of  the  major  contributors.  Most  of 
the  interviewees  felt  that  the  wrong  selection  of  parts  and 
components  contributed  heavily  to  the  problem  associated 
with  design.  A  few  of  the  respondents  mentioned  that  design- 
caused  failures  are  declining.  The  major  area  where  design 
contributes  to  the  gap  is  a  poor  understanding  of  the  real 
operational  environment.  The  other  problems  associated  with 
design  included  the  use  of  design  tools  such  as  Computer- 
Aided  Design  (CAD) ,  which  seldom  consider  R&M  parameters. 

Design:  Possible  Solution.  Several  of  the 

respondents  suggested  that  a  more  robust  design  is  necessary 
to  withstand  the  uncertainties  associated  with  the 
operational  environment.  Taguchi's  method  to  design  was 
mentioned  as  an  alternative. 


Management.  Most  of  the  respondents  who  mentioned 
management  as  one  of  the  major  contributors  to  the 
reliability  gap  ranked  it  as  either  the  most  important 
contributor  or  the  second  most  important  contributor.  The 
average  for  management  is  either  the  highest  or  second 
highest  in  any  set  of  interviews  (see  Tables  4  through  7) . 

It  was  cited  that  most  managers  are  worried  about  short 
term  benefits  rather  than  long-term  gains.  Consequently, 
fundings  in  the  area  of  reliability  were  rarely  sufficient 
since  reliability  takes  time  (one  of  the  findings  from  the 
RCA  report  from  1955;  see  Appendix  A) . 

In  the  area  of  short-term  management,  Ralph  E.  Evans 
(27)  wrote  an  editorial  and  some  of  his  interesting  words  were 


Short-term  management  is  the  major  primary 
bottleneck  to  achieving  the  levels  of  reliability 
that  are  nominally  desired  by  users.  Schedule  and 
cost  are  like  water  and  food — you  can't  live  long 


T.ri  f  V\nn^  +*V»x 


Dril  l  oK  -J  1  A  /  anri 

*'V4-  AW4M  ^  UHM 


VJW41*  J.  4.  V  j  j  X.  UUlvS 


are  like  vitamins — you  develop  the  symptoms  (e.g. 
beri-beri  and  scurvy  for  vitamin  B  &  C 
deficiencies) .  In  large  part,  it  is  the  social, 
economic,  and  political  environments  that  force 
managers  to  become  oriented  to  the  short-term. . . . 


Short-term  managers  are  anxious  and  willing  to 
confuse  form  with  substance.  They  have  neither  the 
time  nor  inclination  to  change  a  company's  way  of 
life.  Instead  they  go  for  1-shot  things  that  have 
lots  of  hoopla  and  show,  like  building  a  quality- 
productivity  center,  insisting  that  everyone 
install  the  trappings  of  STATISTICAL  quality 
control,  and  motivational  slogans  (Quality  Is 
Number  One) ....  Short-term  managers  put  middle 
managers  in  the  double  bind  (damned  if  you  do, 
and  damned  if  you  don't)  by  not  allocating 
sufficient  resources  to  them  for  achieving  the 
lofty  corporate  goals.  If  middle  management  takes 
the  lofty  goals  seriously,  then  it  will  fail  on 
production  or  schedule.  If  it  keeps  to  production 


and  schedule  requirements,  then  it  fails  on  the 
lofty  corporate  goals....  [27] 

In  addition,  the  interviewees  mentioned  that  not  enough 

emphasis  is  put  on  training  engineers  to  design  systems  that 

*  are  robust  enough  to  withstand  the  operational  environment. 

In  the  same  editorial  of  Short-Term  Management,  Ralph  Evans 

mentioned  that: 

Most  people  need  actual  (not  vicarious)  experience 
in  other  jobs  in  order  to  appreciate  them.... 

A  designer  needs  to  experience  first-hand  what  the 
problems  of  manufacturing  and  field-service  are. 

The  manufacturing  engineer  needs  to  experience  the 
frustrations  of  design  and  of  living  with  the  junk 
the  company  ships.  [27] 

Management:  Possible  Solution.  No  specific 

solution  was  cited.  However,  most  respondents  mentioned 
that  somehow  management  has  a  "piece  of  the  action"  in  all 
the  identified  contributors.  Without  proper  managerial 
emphasis,  none  of  the  other  possible  solutions  is  viable.  In 
adc.ition,  Mr.  David  Weber  (General  Electric:  Lead  R&M 
Engineer) ,  who  has  more  than  25  years  of  experience 
in  the  area  of  reliability,  cited  that  top-management 
involvement  is  needed  and  "what  is  also  needed  are  people 
who  can  sustain  themselves  through  the  downs  as  well  as  the 

- -  — .  it 
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Training.  The  respondents  mentioned  four 
categories  of  personnel  who  lack  proper  training.  For  the 
designers,  they  mentioned  that  the  R&M  parameters  are  rarely 
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taught.  In  addition,  the  interviewees  contended  that  the 
designers  do  not  understand  the  manufacturing  processes  and 
field  maintenance  to  design  out  producibility  ard 
maintenance  problems.  For  the  manufacturers,  the  experts 
felt  that  most  of  them  do  not  have  the  experience.  For  the 
maintainers,  the  interviewees  felt  that  because  they  lack 
proper  training,  the  maintainers  themselves  induced  a  lot  of 
the  problems.  And  finally,  for  the  operators,  the 
respondents  concluded  that  operators  may  induce  problems  or 
may  h?  re  different  expectations  about  the  performance  of  the 
equipment/system . 

Training: _ Current  ♦  Mr.  Virgil 

Rehg  (AFIT  Professor:  Quantitative  Methods  &  Statistics) , 
conducted  a  couple  of  surveys,  using  field  people  at  the 
operating  bases,  to  identify  problems  that  prevent  them  from 
doing  their  job.  Quite  a  number  of  people  cited  training,  or 
the  lack  of  it,  prevents  them  from  doing  their  job  better. 
(Professor  Rehg's  surveys  on  "Barriers  to  Quality") . 

Isaininai _ Possible  Solution.  Among  the. 

responses,  a  few  of  the  interviewees  suggested  training 
people  from  a  system's  engineering  approach  of  how  each 
discipline  can  effectively  integrate  vith  the  other. 

Human  Performance.  The  consensus  among  the  experts 
was  that  the  inability  to  predict  human  performance  and 
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motivation,  etc. ,  contributed  significantly  to  the 
reliability  gap. 

Human  Performance;  Possible  Solution.  The 
respondents  mentioned  that  human  engineering  is  a  possible 
approach.  In  addition,  a  few  of  the  respondents  suggested 
that  management  emphasis  can  strongly  influence  human 
performance,  both  positively  and  negatively. 

Software.  The  few  interviewees  who  mentioned 
software  as  one  of  the  major  contributors  to  the  reliability 
gap  classified  most  of  the  software-related  problems  as 
induced.  The  respondents  suggested  that  we  are  still  in  the 
early  stages  of  software  engineering.  No  specific  solution 
was  cited. 

Politics.  All  the  respondents  agreed  that 
Congressional  influences  and  bureaucratic  pressures  are 
unable  to  predict  and  greatly  impact  the  possible  management 
emphasis/actions.  Most  respondents  think  that  management 
emphasis/actions  influence  all  the  other  contributors, 
either  directly  or  indirectly.  No  one  suggested  the 
possibility  of  a  solution  in  this  area. 

Packaging,  Handling,  and  Transportation  fP.K.&T).  ' 
This  is  a  unique  contributor,  mentioned  onxy  by  Industry 
personnel.  Because  of  their  line  of  business,  most  of  their 
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field  problems  are  associated  with  P,H,&T.  No  suggested 
solution  was  mentioned. 


Requirement .  Several  of  the  respondents  felt  that 
the  requirements  are  usually  unclear  and  poorly  defined.  A 
few  of  the  respondents  stated  that  the  requirements  are 
fiscally  controlled.  Rather  than  planning  on  how  to  counter 
the  threat,  oftentimes  it  is  planning  on  "how  much  we  think 
we  can  afford."  Most  of  the  time,  it  was  mentioned  that  the 
stated  requirements  were  not  hard-and-f irm  requirements. 
Consequently,  they  were  easily  overpowered  by  the 
constraints  of  cost  and  schedule. 

Requirement :  Possible  Solution.  In  the  area 

of  realistic  requirement,  Mr.  McCarty  (AFIT  Professor: 

Weapon  System  Management) ,  who  has  over  20  years'  experience 
in  acquisition,  has  been  advocating  for  years  that  there 
should  be  a  concepts  division,  where  concepts  are  analyzed 
in  accordance  with  the  identified  needs.  The  stability  of 
the  people  in  the  "concepts  division"  will  help  establish 
an  ongoing  dialogue  with  the  using  commands,  which  in  turn 
will  help  identify  what's  achievable  and  what's  optimistic 
with  the  given  constraints  of  costs  and  time.  Professor 
McCarty  also  mentioned  that  the  people  in  the  concepts 
division  should  be  divorced  from  the  management  of  the 
development  of  the  program  in  order  to  have  an  objective 
approach  to  the  best  answers  for  the  needs. 
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Phase  III  Findings 


Experts'  opinions,  pertaining  to  Questions  7  and  8  of 
the  Interview  Questionnaire,  were  consolidated. 

Specifically,  opinions  on  reliability  growth  model,  initial 
reliability,  growth  rates,  and  effectiveness  factor  were 
analyzed.  The  data  from  the  interviews  were  collected  and 
analyzed  according  to  the  methodologies  described  in 
Chapter  III. 

Reliability  Growth  Model.  The  question  was:  If  you 
were  to  develop  a  reliability  growth  plan,  what  growth  model 
would  you  use?  Most  of  the  responses  (DoD  &  Industry)  were 
Duane  or  modified  Duane  (i.e.,  AMSAA).  The  reasons  were 
its  simplicity  and  ease  of  use.  Those  who  chose  AMSAA  think 
it  is  more  statistically  accurate.  A  couple  of  suggestions 
were  made  as  to  how  to  choose  the  right  model: 

1.  Use  research  and  computer  simulation  to  find  the 
mort  appropriate  model . 

2 .  Use  Duane  during  the  initial  stages  when  the  data 
are  scarce.  During  the  mid-  and  later  stages,  plot 
the  data,  determine  the  failure  distributions, 
choose  a  model  (parametric  or  nonparametric)  that 
fits  the  data  the  best,  and  perform  a  goodness-of- 
fit  test. 

Initial  Reliability.  The  question  was:  If  you  were  to 
develop  a  reliability  growth  plan,  what  starting  reliability 
value  would  you  use?  Would  you  use  a  starting  reliability 
value  of  10%  of  the  cumulative  mean  time  between  maintenance 
(inherent)?  20%?  Most  of  the  experts/practitioners 
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responded  that  they  would  expect  an  initial  reliability 
close  to  10%.  However,  most  of  them  would  like  to  see  an 
initial  reliability  closer  to  25%  or  30%.  With  a  higher 
initial  reliability,  they  expressed  that  it  does  not  take  as 
long  to  reach  the  mature  stage. 

Reliability  Growth  Rate.  The  question  was:  If  you 
were  to  develop  a  reliability  growth  plan,  what  growtt  rates 
would  you  use  for  the  four  types  of  weapon  systems:  Aircraft, 
Missile,  Spacecraft,  and  Ground-based  complex  electronic 
systems?  The  responses  (averages  and  modes  during  the  Full- 
Scale  Production  Phase  of  the  acquisition  process)  were  as 


follows: 

Missile 

Complex  Ground  Electronics 

Average 

0.30 

0.33 

0.27 

Mode 

0.30 

0.35 

0.30 

Effectiveness  Factor.  The  question  was:  If  you  have 
identified  the  corrective  actions,  what  effectiveness  factor 
would  you  use?  For  example,  100%  effectiveness  factor  means 
the  corrective  action  will  correct  the  identified 
deficiency;  it  will  never  happen  again.  Of  the  38 
interviewees,  25  responded  to  this  question  and  the  average 
and  the  mode  were  both  7Q%. 
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The  findings  for  each  phase  were  presented  separately. 
Table  8  summarizes  the  results  from  each  phase,  in  the  next 
chapter,  the  conclusions  and  recommendations  of  this  study 
will  be  presented. 

Table  8.  Summary  of  Findings 


Phase  Findings 


I  Reliability  gap  exists  in  airborne  tactical 
radars  for  both  Air  Force  and  Navy  aircraft. 
Figure  4 . 

II  The  most  significant  contributors  to  the 
reliability  gap  were  identified  and  rank 
ordered.  See  Tables  4,  5,  6,  and  7. 

Ill  Reliability  growth  responses  were 
consolidated: 

a.  Most  popular  growth  model: 

- Duane  or  modified  Duane  (i.e.,  AMSAA) 

b.  Initial  reliability: 

- Been  experiencing:  Close  to  10% 

---Would  like  to  see:  25%  to  30% 

c.  Growth  rate:  During  Full-Scale 

Production 
(average,  mode) 

- Aircraft:  (0.30,  0.30) 

- Missile:  (0.33,  0.35) 

— -Complex  Ground 

Electronics:  (0.27,  0.30) 
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V.  Conclusions  and  Recommendations 


Introduction 

This  chapter  summarises  the  results  of  this  study  and 
makes  recommendations  for  future  studies .  This  chapter 
begins  with  a  review  of  this  research  followed  by  a 
comparative  discussion  between  the  findings  from  this  study 
and  those  from  the  literature  review,  and  concludes  with 
recommendations  for  future  studies. 

Review 

Over  the  years,  the  difference  (gap)  between  factory 
(whether  specified,  predicted,  or  demonstrated)  and  field 
reliability  has  been  documented  in  numerous  studies. 
Historically,  the  gap  in  avionics  equipment  was  reported  to 
range  from  7:1  to  20:1  (38:231). 

The  overall  objective  of  this  research  is  to  investigate 
why  the  actual  field  reliability  is  consistently  different 
(usually  lower)  than  the  factory  reliability.  This  research 
begins  with  an  exhaustive  literature  review.  First  on  the 
definition  of  reliability,  then  the  development  of 
reliability  engineering,  and  finally,  the  problems  of 
unreliability,  all  to  set  the  stage  for  the  discussion  of 
the  research  problem — the  reliability  gap.  This  inability 
to  accurately  relate  factory  reliability  to  the  field 
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reliability  has  a  significant  impact  on  operational 
readiness  and  life  cycle  costs.  In  order  to  better 
understand  the  problem,  traditional  ways  (along  with  their 
limitations)  of  assessing  reliability  and  some  practical 
questions  and  answers  associated  with  the  reliability  gap 
were  examined. 

To  recapitulate,  the  objectives  of  this  study  are: 

1.  To  examine  the  existence  of  the  reliability  gap  in 
airborne  tactical  radars  of  Air  Force  and  Navy 
aircraft. 

2.  To  identify  and  analyze  the  major  contributors  to 
the  reliability  gap.  Specifically,  to  identify  the 
most  significant  contributors. 

3.  To  explore  ways  of  minimizing  the  reliability 
gap.  Specifically,  to  examine  some  practical 
guidelines  on  reliability  growth. 

To  accomplish  the  first  objective,  tactical  airborne 
radars  were  examined.  The  courtesy  slide  from  NAVAIR 
indicated  that  the  reliability  gap  for  tactical  airborne 
radars  existed  in  both  Air  Force  and  Navy  aircraft.  To 
accomplish  the  second  objective,  personal  as  well  as 
telephone  interviews  were  conducted.  Experts/ 

Practitioners  were  asked  to  rank  order  the  major 
contributors  to  the  reliability  gap.  In  addition,  a  second 
round  of  interviews  was  conducted  to  validate  the  findings 
of  the  first  round.  The  specific  findings  were  documented 
in  Chapter  IV.  To  accomplish  the  third  objective,  experts'/ 
practitioners'  opinions,  various  studies  conducted  by  the 
experts,  and  appropriate  historical  findings  were  analyzed. 
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Conclusion:  Comparative  Discussion 


This  portion  of  the  chapter  compares  the  findings  from 
this  study  to  those  from  the  literature  review.  It  follows 
the  same  format  as  the  previous  chapters;  it  is,  divided 
into  three  distinct  phases,  each  corresponding  to  the 
appropriate  objective  of  this  study. 

Phase  I.  The  reliability  gap  existed  in  airborne 
tactical  radars  for  both  Air  Force  and  Navy  aircraft. 

Phase  II.  The  major  contributors  identified  in  this 
study  were  compared  to  the  factors  found  in  the  literature 
review.  By  comparing  Table  2  (findings  from  literature 
review)  and  Tables  4  through  7,  the  results  were  found  to  be 
comparable.  The  additional  knowledge  gained  in  this  study 
are: 

1.  The  relative  importance  of  these  major  contributors. 

2.  The  different  perspectives  between  DoD  and  Industry 
personnel  on  these  contributors. 

Phase  Ilf.  Various  approaches  were  mentioned  on  how  to 
minimize  the  reliability  gap.  The  responses  to  Question  6 
of  the  Thesis  Questionnaire  addressed  the  kinds  of 
incentives  for  contractors  to  produce  more  reliable 
products.  See  Appendix  C,  Question  6.  Questions  7  and  8  in 
Appendix  C  address  some  practical  guidelines  to  reliability 
growth.  Realizing  that  reliability  growth  is  not  the 
panacea  to  the  reliability  gap  problem,  the  reliability 
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growth  program  does,  however,  provide  some  guidelines  on  how 
to  minimize  the  gap.  In  this  phase,  some  of  the  practical 
guidelines  cited  by  the  experts/practitioners  on  reliability 
growth  in  the  following  areas  were  collected: 

lr  Reliability  growth  model 

2.  Initial  reliability 

3 .  Growth  rate 

4.  Effectiveness  factor 

Reliability.  Growth  Model.  From  the  literature 
review,  three  prominent  studies  on  reliability  model  were 
cited.  The  analysis  for  identifying  the  optimal  reliability 
growth  model  is  beyond  the  scope  of  this  study;  however, 
experts '/Practitioners '  opinions  were  gathered  on  the  kind 
of  reliability  growth  models  that  are  being  used  in  the 
field.  The  similarity  between  the  findings  from  the 
literature  review  and  this  study  both  concluded  that  the 
Duane  or  the  modified  Duane  (i.e.,  AMSAA)  is  the  most 
popular  model. 

Initial  Reliability.  From  the  literature  review, 
the  Army  identified  that  most  "successful  programs  usually 
begin  with  an  initial  MTBF  of  at  least  1/3  or  1/4  of  the 
desired  MTBF"  (6:211) .  Though  most  experts/practitioners 
have  been  experiencing  an  initial  reliability  of  less  than 
10%  of  the  desired  reliability,  most  of  them  would  like  to 


107 


have  an  initial  reliability  25%  to  30%  of  the  desired 
reliability. 

Growth  Rates.  From  the  literature  review,  the 
growth  rates  were  cited  to  range  from  0.35  to  0.5  (45:78). 
From  this  study.  Table  S  summarizes  the  practical  growth 
rates  for  each  type  of  weapon  system  during  full-scale 
production  of  the  acquisition  phase.  To  some  extent,  the 
ranges  for  the  growth  rates  may  be  comparable;  however,  the 
highest  rate  (mode)  identified  by  the  experts/practitioners 
in  this  study  was  0.35. 

It  is  important  to  understand  that  the  risks  of  the 
reliability  gap  can  be  minimized  if  the  uncertainties 
associated  with  reliability  growth  are  recognized.  The 
recommended  growth  rates  for  the  weapon  systems  can  be  used 
as  a  guide  for  planning  purposes.  With  the  findings  of  this 
study,  it  is  hoped  that  managers  will  realize  that  there  are 
tremendous  amount  of  risks  involved  in  growth  rates  that  are 
greater  than  0.5.  In  fact,  a  few  experts/practitioners 
commented  that  during  their  career  (average  of  fifteen 
years),  ”,..0.4  was  the  fastest  that  I've  seen  anything 
grow . " 


Effectiveness  Factor.  From  the  literature  review, 
the  Army  identified  that  most  failure  modes  achieved  a  70% 
fix  effectiveness  factor  (6:211).  Some  of  the  experts/ 
practitioners  are  aware  of  the  Army's  study  on  effectiveness 
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factor  and  also  cited  a  70%  effectiveness  factor.  Others 
cited  the  effectiveness  factor  to  be  higher  than  70%  and 
some  cited  lower.  However,  it  is  interesting  to  note  that 
among  the  25  out  of  the  38  interviewees  (some  did  not 
respond  to  this  particular  question) ,  the  average  and  the 
mode  were  both  70%. 

Several  interviewees  expressed  that  the  risks  of  tha 
reliability  gap  can  be  minimized  if  it  is  recognized  that 
the  fix  effectiveness  for  the  failure  modes  is  not  always 
100%  effective.  As  expressed  by  Dr.  Crow,  the  expert  on 
reliability  growth,  during  the  interview:  "...in  order  to 
do  growth  testing  properly,  you  must  insert  tha  fix  before 
you  continue  testing  because  you  cannot  make  the  assumption 
of  100%  effectiveness  of  any  fix!" 

Recommendations 

This  study  provides  some  conceptual  ideas  on  how  the 
reliability  gap  can  be  minimized  in  order  to  maximize 
operational  readiness  and  optimize  life  cycle  costs. 
Recommend  that  continual  efforts  (i.e.,  a  "rule  of  thumb" 
handbook)  be  made  along  the  lines  of  practical  guidelines  in 
reliability  prediction  techniques  and  reliability  growth 
management . 

The  following  recommendations  are  based  on  the  findings 
from  the  literature  review  and  the  results  from  this  study. 
Like  the  conclusions  of  this  study,  it  is  divided  into  three 
distinct  phases. 
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I 

Phase  I.  In  order  to  minimize  the  gap,  recommend  a 
closer  analysis  of  the  user's  requirement  (to  determine 
whether  it  is  realistic) ,  the  prediction  methodology  (with 
the  limitations  of  MIL-STD-217  in  mind) ,  the  simulated 
environment  during  testing  (recognizing  the  difference 
between  laboratory  vs  operational  environment  and  the  unique 
characteristics  of  each) ,  and  field  data  collection  system. 

In  the  area  of  development  of  the  initial  predicted 
reliability,  recommend  that  the  prediction  methodology  be 
refined  with  field  data.  It  is  interesting  to  note  that 
most  of  the  interviewees  from  the  industry  sector  cited  that 
the  use  of  field  data  is  the  best  possible  information  for 
refining  the  prediction  methodology.  By  using  the  field 
data,  the  uncertainties  associated  with  the  performance  of 
the  equipment  in  the  operational  environment  can  be 
minimized. 

In  the  area  of  the  actual  field  reliability  data, 
recommend  using  the  right  people  (preferably  R&M 
engineers)  to  analyze  the  collected  data,  to  identify  the 
problem  areas,  and  to  recommend  the  corrective  action (s) . 

Phase  II.  Several  major  contributors  have  been 
identified  and  rank  ordered  by  this  study.  In  order  to 
better  understand  the  various  contributors,  recommend 
further  analysis  of  these  major  contributors. 

This  study  summarized  the  findings  on  a  linear  scale. 
This  is  not  to  suggest  that  the  relationships  among  the 
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major  contributors  are  linear.  In  fact,  the  effects  of  some 
contributors  have  been  suggested  to  have  a  significant 
impact  on  the  other  contributors.  The  overlapping  effects 
of  the  contributors  are  beyond  the  scope  of  this  study.  In 
order  to  further  investigate  how  each  of  the  contributing 
factors  influences  the  others,  recommend  that  a  formalized 
factor  analysis  be  conducted.  In  addition,  recommend  the 
major  contributors  be  classified  into  controllable  and 
uncontrollable  contributors  by  organization.  Under  each 
organization,  resources  ought  to  be  spent  only  in  areas 
where  the  contributor (s)  can  be  controlled  by  that 
organization. 

Phase  III.  Various  approaches  have  been  identified  on 
how  to  minimize  the  reliability  gap — specifically,  to 
identify  some  practical  guidelines  on  the  type  of 
reliability  growth  model,  the  initial  reliability,  growth 
rates,  and  effectiveness  factor.  Most  of  the  experts/ 
practitioners  believe  in  growing  (improving)  reliability, 
but  they  also  cautioned  on  the  blind  application  of 
reliability  growth  technique.  The  recommended  course  of 
action  is  to  assess  each  weapon  system  on  a  case-by-case 
basis.  Good  practices,  such  as  applying  the  environment 
correctly,  designing  the  weapon  system  robustly,  assigning 
the  growth  rate  reasonably,  and  testing  for  failures 
effectively,  are  recommended  approaches  in  minimizing  the 
reliability  gap. 
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Appendix  A:  Selective  Summary  of:  Proceedings  of 
the  Conference  on  "The  Reliability  of 
Military  Electronic  Equipment,"  RCA, 
August  1955 


1.  RELIABILITY  IS  A  "SYSTEMS"  PROBLEM! 

It  involves  requirement,  specifications,  design,  test, 
quality  control,  and  "feedback"  to  mention  a  few  of  the 
major  aspects, 

2.  RELIABILITY  REQUIRES  ORGANIZATION! 

The  objective  of  adequate  reliability  cannot  be  achieved 
by  happenstance. .This  dictates  the  need  for  an 
internal  company  organization  whose  sole  interest  is 
reliability. 

3.  RELIABILITY  SETS  A  NEW  STANDARD! 

To  achieve  adequate  reliability,  we  must  establish  a  new 
standard  for  component  and  end- item  specifications; 
similarly,  we  must  establish  a  new  standard  for  design 
criteria,  quality  control,  and  test  procedures ....  One 
might  say  that  this  whole  area  is  the  very  cornerstone 
on  which  industry  and  the  military  must  build  if  we  are 
to  be  successful  in  attaining  our  goal. 

4.  RELIABILITY  REQUIRES  ADEQUATE  LOGISTIC  SUPPORT! 

This  involves  more  than  the  timely  supply  of  spare  parts 
and  test  equipment  to  the  user;  the  reliability  or 
replacement  parts  must  equal  that  of  the  original 
components . 

5.  RELIABILITY  NECESSITATES  ADEQUATE  USER  TRAINING! 

The  designer  can  simplify  training  requirements  through 
modular  construction  and  so  on,  but  user  personnel  must 
attain  an  adequate  level  of  training  if  the  full 
benefits  of  improved  apparatus  reliability  are  to  be 
realized. 

6.  RELIABILITY  TAKES  TIME! 

As  a  generalization,  it  might  be  said  that  the  built-in 
reliability  is  proportional  to  the  time  allowed  the 
supplier  to  design,  debug,  produce  a  pilot  run,  and 
incorporate  changes  or  improvements  in  production  as  a 
result  of  "feedback"  from  field  experience. .. .time  is 
the  mortar  which  binds  this  whole  structure  of 
reliability  together.... 

7.  RELIABILITY  REQUIRES  MILITARY-INDUSTRY  TEAMWORK. 
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Appendix  B:  Thesis  Questionnaire, 


Position: 

Organization:  Years  of  Experience  in  Reliability: 

Areas  (reliability  related)  of  Interest: 

Mailing  Address: 


THESIS  TOPIC:  Comparison  between  the  theoretically 

predicted  reliability  values  vs  the  actual  field  data  of 

weapon  systems. 

QUESTIONS : 

1.  What  are  the  major  contributors  to  the  difference 
between  the  theoretically  predicted  values  and  the 
actual  field  data? 

2 .  Why  do  systems  fail? 

3.  Are  most  failures  related  to  poor  design?  Explain  why. 

4 .  Are  most  failures  related  to  poor  manufacturing 
processes?  What  are  the  bottlenecks  in  the 
manufacturing  processes?  Explain  why. 

5.  From  your  personal  experience,  what  is  the  distribution 
of  the. major  contributors  mentioned  in  Question  1?  If 
your  personal  experience  is  limited,  your  educated 
guesses  will  suffice.  If  you  feel  uncomfortable 
assigning  percentages,  please  rank  order  the  major 
contributors . 

6.  What  kind  of  incentives  should  the  government  pursue  in 
order  to  motivate  contractors  to  produce  more  reliable 
products? 

7.  If  you  were  to  develop  a  reliability  growth  plan,  what 
growth  model  would  you  use?  What  growth  rate  would  you 
use?  What  reliability  starting  point  would  you  use? 
Would  £ou  use  a  starting  reliability  value  of  10%  of 
the  cumulative  Mean  Time  Between  Maintenance  (Inherent)? 
20%?  Please  answer  the  questions  for  the  four  types  of 
weapon  systems:  Aircraft,  Missile,  Spacecraft,  and 
Ground-based  complex  electronic  systems. 

8.  If  you  have  identified  the  corrective  actions,  what 
effectiveness  factor  would  you  use?  For  example,  100% 
effectiveness  factor  means  the  corrective  action  will 
correct  the  identified  deficiency;  it  will  never  happen 
again. 
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9. 


What  data  collection  system  would  you  use?  What  are  the 
problems  with  our  current  data  collection  system?  What 
changes  would  you  recommend? 

10.  If  you  only  have  enough  resources  to  choose  one  of  the 
two,  which  one  would  you  choose?  Explain. 

a.  Reliability  demonstration  vs  Reliability  growth 

b.  Environmental  Stress  Screening  (ESS)  vs  Reliability 
growth 

Please  feel  free  to  call  Katherine  Ma,  AV  785-5435,  if  you 
have  any  questions.  Would  like  the  initial  responses  back 
by  15  Apr  88.  Thanks! 
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Appendix  C:  Thesis  Questionnaire  Summary 


THESIS  TOPIC:  Comparison  between  the  theoretically 
predicted  reliability  values  vs  the  actual  field  data  of 
weapon  systems. 

OBJECTIVES : 

- To  identify  the  major  contributors  to  the  reliability  gap 

between  the  predicted  vs  the  actual  field  data 

- To  examine  ways  of  minimizing  the  gap  (Questions  1  to  5) 

- To  rec^punend  some  guidelines  (experts'  point  of  view)  for 

an  effective  reliability  program  (Questions  6  to  10) 

During  the  interviews,  I  asked  the  experts  to  rank  order 
the  major  contributors  to  the  reliability  gap  between  the 
predicted  vs  the  actual  field  data.  I  then  use  the  ranking 
methodology,  which  I  will  describe  in  detail  in  my  thesis, 
to  rank  order  (from  highest  to  lowest)  all  the  major 
contributors  accordingly.  The  following  is  a  summary  of  the 
result. 

The  result  is  based  <fn  the  first  30  people  I  have 
interviewed  (personal  &  telephone) .  In  order  to  give  a  more 
definitive  approach  to  my  thesis,  I  have  taken  the  privilege 
of  grouping  the  major  contributors  into  controllable  and 
uncontrollable  variables.  The  criteria  for  determining  what 
is  controllable  and  what  is  uncontrollable  are  totally 
subjective  on  my  part.  My  reasoning  for  classifying  a 
variable  as  controllable  is  that,  through  the  course  of  my 
literature  research  and  interviews,  I  have  learned  of  a 
feasible  solution  on  how  to  control  that  variable,  and  thus 
minimize  the  gap  between  the  predicted  vs  the  actual  field 
data.  The  uncontrollable  variables  are  just  that,  variables 
that  cannot  be  controlled  and  which  may  have  a  significant 
impact  on  all  of  the  controllable  variables.  If  you  do  not 
agree  with  my  classification,  please  let  me  know. 

In  order  to  have  a  clear  understanding  of  what  each 
major  contributor  represents,  I  have  included  the  responses 
from  my  interviews  on  the  following  pages. 


Controllable  Uncontrollable 

Variables  Variables 


I. 

DATA 

I. 

TOP  MGMT  COMMITMENT 

II. 

MANUFACTURING  PROCESSES 

• 

H 

H 

ENVIRONMENT 

III. 

DESIGN-RELATED 

III. 

ACQUISITION  PROCESS 

IV. 

REQUIREMENTS 

IV. 

POLITICS 

V. 

PREDICTION  TECHNIQUES 

V. 

HUMAN  PERFORMANCE 

VI. 

TRAINING 

MAJOR  CONTRIBUTORS  TO  THE  RELIABILITY  GAP:  (Questions  i-5) 
Controllable  Variables: 


I.  DATA: 

- COLLECTION: 

- Inaccurate,  untimely,  and  incomplete 

- Lack  of  motivation,  training,  and  emphasis 

- DEFINITION: 

- Classification  of  failures:  relevant  vs 

nonrelevant  (3)* 

- Scoring  procedures 

- Contractual  vs  operational  (2) 

- Flying  vs  operating  hours  (2) 

- Unclear  R&M  terms,  >60  R&M  terms  (2) 

- Reliability  vs  performance  problems 


II.  MANUFACTURING  PROCESSES: 

- Using  processes  that  are  not  yet  proven 

- The  processes  are  becoming  so  microscopic  that  it 

is  difficult  to  control  and  inspect 
- Quality  of  the  components 

— -MIL-HDBK  predictions  do  not  take  manufacturing 
processes  into  account 

- Factory  workmanship  (training  level) 

- Producibility :  Design-related 

- Proper  design  can  absoro  a  lot  of  the  problems 

- Quality: 

- Contractor-dependent 

- Does  a  total  quality  management  program  exist 


III.  DESIGN-RELATED: 

- Immature  technology:  For  example.  Very  High  Speed 

Integrated  Circuit  (VHSIC)  technology 

- Mistakes  in  parts  and  components  selection  (3) 

- Suboptimization 

- Integration 

- Complexity 

- Constrained  by  time  and  money 

- Design  tool:  CAD  rarely  include  R&M  parameters 

- Software :  Induced  failures 


*  Note:  The  number  in  parentheses  indicates  the  number  of 
experts  who  gave  the  same  response. 
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IV.  REQUIREMENTS: 


- Unclear,  not  well  defined  (3) 

- Requirements  are  NOT  hard,  firm  requirements 

- Fiscally  constrained 

- Concession  we  make  to  reality,  in  terms  of  cost  and 

schedule 


V.  PREDICTION  TECHNIQUES: 

- MIL-STD-217  prediction: 

- Lack  of  predictors'  failure  rates  for  hydro¬ 
mechanical  and  interconnectors  and  other  new 
technologies  (2) 

- Assume  perfect  design  and  manufacturing 

processes  of  components  (4) 

- Mostly  are  component-level  testings 

- Paper  design,  all  analytical 

- Too  many  unknown  variables  which  are  average  of 

various  sources  (3) 

- Some  predictors  are  engineers'  opinions 

- Predictors  seldom  match  the  operational 

predictions 

- Misapplication  of  the  predictors 

- Inaccurate  due  to  using  nonoperational  type 

environment  (4) 

- No  standard  definitions 

- Do  not  address  interconnections 

- Assume  perfect  accessibility  for  maintainers 

(one-deep)  (2) 

- Improper  use  of  reliability  model 

- Source  selection  criteria  force  the  contractors  to 

always  put  their  best  foot  forward  upfront 

- Mentality  factor:  Prediction  as  a  box  filling 

exercise 

- Assume  no  operator  or  maintainer-induced  failures 


VI.  TRAINING: 

a.  Designers: 

- Trained  to  design  things  to  perform  rather  than 

design  things  not  to  fail. 

- R&M  parameters  rarely  taught 

- Designers  do  not  understand  field  maintenance  in 

order  to  design  out  maintenance  problems 

- Designers  do  not  understand  manufacturing 

processes  to  design  out  or  make  the  processes 
more  producible 
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b.  Maintainers: 

- Misuse  of  equipment  due  to  lack  of  training 

- Induced  problems  due  to  lack  of  experience 

- Field  personnel  not  as  experienced  as  laboratory 

engineers 

- Lack  of  job  proficiency  during  transfer 

- Lots  of  variables  in  OJT 

c.  Operators: 

- Improper  air  crew  expectations  of  the  performance 

of  certain  pieces  of  equipment  may  induce 
additional  write-ups 
- Operators- induced  failures 

d.  Manufacturers: 

- Lack  of  experience 


Uncontrollable  Variables: 

I.  TOP  MANAGEMENT  COMMITMENT: 

- Program  management 

- Program  managers  (PMs)  rarely  include  any 

funding  for  reliability  growth 

- Return  on  investment  is  too  long  to  have  an 

impact  on  PM's  tour 

- Lack  of  emphasis  on  training  engineers  to  design 

things  not  to  fail,  rather  than  to  design  things 
just  to  perform 

- Lack  of  emphasis  on  designs  that  are  robust  to  the 

environment 


II.  ENVIRONMENT: 

- Mismatch  of  operational  environment  to  the  design 

envelope  (10) 

- Misapplication  of  equipment 

- Change  in  operational  scenario  intentionally  or 

unintentionally 

- We  do  not  understand  the  operational  environment  (9) 

- Difficult  to  predict  something  we  do  not 

understand 

- Inadequate  field-type  testing  to  find  out  more 

about  the  operational  environment 

- Too  expensive  to  duplicate  the  operational 

environment 
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- Operational  environment  is  harsher  than  the 

laboratory  environment 

- Lack  of  good  translational  factor  between 

operational  and 
laboratory  environment 

- Lack  of  environmental  screening 

- Nonapplication  of  MIL-STD-781C: 

- Design  is  not  robust  to  the  operational 

environment 

- System  integration:  Interactions  of  components  (2) 

- GFE  interfaces  with  other  equipment 

- MIL— HDBK  predictors  have  a  lot  of  uncertainty 

- Corrosion  control  not  taken  into  consideration 

- Subjective  judgment: 

- Lead  engineers  gut  feel  on  new  improved 

technology 


III.  ACQUISITION  PROCESS: 

- Process  too  long  to  accommodate  the  changing  threat 

- Process  too  short  for  adequate  development  of 

certain  new  technology 

- Insufficient  funds  and  time 

- Compromises  we  made  throughout  the  process 

— -Contradictory  approach:  Lowest  bidder  wins  the 
contract  and  yet  we  want  high-quality  components 


IV.  POLITICS: 

- Bureaucratic  pressure 

- Cost  and  schedule 

- Congressional  influences 


V.  HUMAN  PERFORMANCE: 

- Humans  are  more  prone  to  make  mistakes  under 

stressful  conditions 

- Inability  to  predict,  level  of  effort,  motivation, 

etc.,  of  the  following  groups  of  people: 

a.  Designers 

b.  Maintainers: 

- Maintenance  practices 

- Triggering  of  equipment 

- Induced  errors 

- False  removals 

- Support  equipment 
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c.  Operators: 

- Don't  fully  understand  the  performance  of  the 

equipment 

d.  Manufacturers 

e.  Contractors: 

- Contractors'  integrity  to  tell  us  all  the 

problems.  Sometimes  we  do  not  find  the 
problems  until  the  systems  are  fielded 

- Contractors,  like  our  designers,  know  how  to 

design  to  performance,  but  not  to  reliability, 
quality,  and  producibility 
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THESIS  QUESTIONNAIRE:  Question  6 


What  kind  of  incentives  should  the  government  pursue  in 
order  to  motivate  contractors  to  produce  more  reliable 
products? 

RESPONSES : 

I.  Positive  and  negative  incentives:  (20) 

- Convincing  the  contractors  that  the  customers  are 

serious 

- Show  the  contractors  that  the  government  is  ready  to 

pay  R&M  upfront 

II.  Monetary:  (18) 

- Purely  profit 

III.  Need  a  workable  approach:  (10) 

a.  Specify  a  mutually  (contractor  &  government) 
acceptable  parameter  that  is  measurable.  The 
parameter  (i.e.,  Mean  Time  Between  Demand)  has  to 
be  based  on  operational  data. 

b.  Need  an  accurate  data  collection  system  that  is 
acceptable  to  both  the  contractor  and  the 
government . 

c.  Conduct  data  review  board  to  discuss  data  accuracy. 
The  data  review  will  be  attended  by  both  contractor 
and  government  representatives. 

IV.  Warrant  against  operational  measures  and  operational 
environment 

- Use  meaningful  terms  to  tell  the  contractor  what 

the  government  really  needs. 

V.  Use  stepwise  incentives  with  time  constraints 

VI.  Use  competition  to  tie  to  production  or  follow-on 
contracts 

VII.  Ensure  the  contract  is  well-written  so  contractors 
cannot  fail  to  perform.  For  example,  specify  R&M 

as  a  separate  item  in  the  Request  For  Proposal  (RFP) . 

VIII.  Need  to  convince  contractors  that  there  will  be  more 
money  involved  in  producing  more  reliable  products 
than  in  just  selling  spares. 

IX.  Government  should  buy  more  mature  systems. 

X.  Encourage  contractors  to  continue  reliability  growth 
programs  past  the  end  of  DT&E  and  until  maturity. 
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THESIS  QUESTIONNAIRE:  Question  7 

If  you  were  to  develop  a  reliability  growth  plan,  what 
growth  model  would  you  use?  What  growth  rate  would  you  use? 
What  reliability  starting  point  would  you  use?  Would  you 
use  a  starting  reliability  value  of  10%  of  the  cumulative 
Mean  Time  Between  Maintenance  (Inherent)?  20%?  Please 
answer  the  questions  for  the  four  types  of  weapon  systems: 
Aircraft,  Missile,  Spacecraft,  and  Ground-based  complex 
electronic  systems. 

RESPONSES: 

RELIABILITY  GROWTH  MODEL:  The  most  popular  model  is  Duane, 
or  modified  Duane  (i.e.,  AMSAA) . 

Other  suggestions  included: 

- Do  research  and  computer  simulation  to  find  the  most 

appropriate  model 

- Use  Duane  during  the  initial  stages.  During  the  mid-  and 

later  stages,  plot  the  data,  determine  the  failure 
distributions,  choose  a  model  (parametric  or  non- 
parametric)  that  fits  the  data  best,  and  perform  a 
goodness-of-fit  test 


RELIABILITY  GROWTH  PATE:  A  function  of  redesign  and 
manufacturing  corrections.  Highly  dependent  on  management 
attention.  A  couple  of  experts  expressed  that  "0.4  was  the 
fastest  that  I've  seen  anything  grow." 


FSD 

PRODUCTION 

* 

NEW  DESIGN 

NEW  DESIGN 

Aircraft 

0.26 

0.1 

Missile 

0.28 

— 

Spacecraft 

0.43 

— 

Avionics 

??? 

??? 

Engines 

??? 

??? 

Airborne  computers 

??? 

??? 

Complex  Grd.  electronics 

0.24 

??? 

Mechanical 

??? 

??? 

Structural  (applicable?) 

??? 

??? 

INITIAL  STARTING  POINT: 

Cumulative  MTBM  (I) 

- Mostly  have  been  experiencing:  <10% 

- Would  like  to  see:  25-30% 
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THESIS  QUESTIONNAIRE:  Question  8 

If  you  have  identified  the  corrective  actions,  what 
effectiveness  factor  would  you  use?  For  example,  100% 
effectiveness  factor  means  the  corrective  action  will 
correct  the  identified  deficiency;  it  will  never  happen 
again. 

RESPONSES: 

Overall  %  =  70% 

Other  %: 

- Design:  85% 

- Manufacturing  processes:  73% 

- Human  factors  and  others:  70% 

- Avionics:  73% 

- Mechanical:  98% 

- Electronics  and  structural:  73% 


A  few  of  the  experts  expressed  that:  "Discount  failures 
with  identified  corrective  actions  is  a  great  idea  if  the 
corrective  actions  work.  In  most  cases,  the  implementation 
of  the  identified  corrective  actions  takes  too  long  and  it's 
too  costly." 


In  response  to  the  above  comments,  I  have  asked  a  few  experts 
the  following  questions.  Please  provide  your  answers  so  I 
can  get  a  larger  sample  size. 

1.  How  long  and  how  much  does  it  cost  to  identify  a 
corrective  action? 


2 .  How  long  and  how  much  does  it  cost  to  do  the  failure 
analysis? 


3 .  How  long  and  how  much  does  it  cost  co  implement  the 
corrective  action? 
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THESIS  QUESTIONNAIRE:  Question  9 

What  data  collection  system  would  you  use?  What  are  the 
problems  with  our  current  data  collection  system?  What 
changes  would  you  recommend? 

RESPONSES : 

For  the  first  group  of  experts  (the  ones  I  interviewed  at 
the  First  Reliability  Growth  Conference  held  in  Boston,  7-9 
March  1988) ,  I  didn’t  ask  you  this  question.  If  any  of  you 
would  like  to  add  to  the  following  set  of  responses,  please 
feel  free  to  do  so. 

The  choices  of  the  data  collection  systems  are:  Maintenance 
Data  Collection  (MDC) ,  Maintenance  Operations  Data  Access 
System  (MODAS) ,  Core  Automated  Maintenance  System  (CAMS)  (as 
advertized),  F-16's  Centralized  Data  System  (CDS), 
Comprehensive  Engine  Management  System  (CEMS) ,  and  System 
Effectiveness  Data  System  (SEDS) .  The  most  popular  choice 
was  F-16's  CDS.  Some  experts  would  like  to  hire  a 
contractor,  and  yet  others  would  like  to  develop  their  own 
data  collection  system. 

The  most  cited  problems  with  our  current  systems  are: 

- Inaccurate,  incomplete,  and  untimely 

- No  motivation  for  data  collectors  to  do  better 

- Lack  of  feedback  system  for  better  data 

- AF^TO  Form  349s  are  used  more  as  man-hour  accounting 

system  than  anything  else 

- Logistical ly  driven,  not  design  driven 

- No  information  on  what  piece/part  failed? 

How  it  failed?  What  caused  the  failure? 


124 


THESIS  QUESTIONNAIRE:  Question  10 
* 

If  you  only  have  enough  resources  to  choose  one  of  the  two, 
which  one  would  you  choose?  Explain. 

a.  Reliability  demonstrations  vs  reliability  growth 

b.  Environmental  Stress  Screening  (ESS)  vs  reliability 
growth 


RESPONSES : 

a.  With  limited  resources,  most  experts  favored 
reliability  growth.  They  think  it  will  provide  the 
most  payback. 

b.  Most  experts  think  that  in  order  to  have  an  effective 
reliability  program,  both  ESS  and  reliability  growth 
are  needed.  When  limited  resources  are  emphasized, 
most  experts  chose  reliability  growth  over  ESS,  but 
most  of  them  expressed  that  it's  not  an  either-or 
situation.  In  fact,  one  expert  told  me  that  if  he  had 
to  choose  between  the  two,  he  would  QUIT  his  job  first! 


125 


Appendix  D:  List  of  Interviewees  &  Their  Organizations 


Department  Of  Defense  (DoD) 

Abrams,  Lt  Col  Fred  L.  Director  Tactical  Logistics 


AFLC  LOC/TL 

Akhbari,  Hamid  Product  Assurance  Engineer 

B-1B  SPO,  AS D/B1ESI 

Andrews,  Capt  Richard  A.  AFIT  Instructor 


Arnold,  Gary  M. 
Babcock,  Paul  F. 

Campbell,  Capt  Clint 


AFIT/LSY 

R&M  Engineer 
F-16  SPO,  ASD/YPEX 

Electronics  Engineer,  GS-12 
Department  of  the  Navy 
Navy  Space  Program  Detachment 

Instructor  of  Quantitative 
Methods  &  Statistics 


Edwards,  Jerry  L. 
Ellner,  Paul  PhD. 


AFIT/LSQ 

Product  Assurance  Engineer 
F-15  SPO,  ASD/VFES 

Director  USAMSAA 


Farr,  Major  Michael 


Fleeger,  Major  (Ret.) 


AMXSY-RM 

Director,  Graduate  Contracting 

Management  Program 

AFIT/LSY 

Product  Assurance  Services 
Consultant 


Hartman,  Lt  Col  Roger  Chief,  Logistics  Studies  and 

Analysis 

AF0TEC/LG4 

Keller,  Major  Fred  D.  AF-1  Deputy  PM  for  Logistics 

Director  of  Transport  and 
Trainer  Programs  SPO 
ASD/SDCBL 


Lapp ,  John 


SCADC  Program  Manager 
ASD/AEAA 
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LaSala,  Kenneth  P.  Chief,  R&M  Division 

Product  Assurance  Engineering 
HQ  AFSC/PLER 

Lilius,  Walter  A.  U.S.  AMETA 

AMXOM-QA 

Mangan,  Tom  Product  Assurance  Engineer 

Propulsion  SPO 
ASD/YZEX 

Mauldin,  Colonel  Thad  Aircraft  Systems  Division 

Directorate  of  Maintenance  and 
Supply 
HQ  USAF 
DCS/L  &  E 

Miller,  Major  Phillip  E.  Director  of  Graduate  Programs 

AFIT/LSG 

Morris,  Seymour  F.  Electronic  Reliability  Engineer 

RADC/RBER 

McCarty,  Dyke  Professor  of  Weapon  System 

Management 

AFIT/LSY 

Olson,  Glenn  R&M  Engineer 

AFALC/OA-OL 

Paige,  Lt  Col  Alan  (Ret.)  Directorate  of  R&M 

and  Evaluation 
AFALC/ERR 

Rehg,  Virgil  Professor  of  Quantitative 

Methods  &  Statistics 
AFIT/LSQ 

Robinson,  Capt  David  G.  Assistant  Professor 

AFIT/ENY 

Rostokowski,  Frank  R&M  Engineer 

Department  of  the  Navy 
Naval  Air  Systems  Command  HQ 

Trakas,  Robert  Head,  R&M  Branch 

Department  of  the  Navy 

Widenhouse,  Carrol  Assistant  Professor  of 

Quantitative  Methods  &  Statistics 
AFIT/LSQ 
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Williams,  Ben,  PhD 


Wlazlo,  Capt  Tom 
Young,  David 
Industry 

Chenoweth,  Halsey  B. 
Crow,  Larry,  H.,  PhD 

Gibson,  Gregory  J. 

Healy,  John  D. 

Horne,  Robin  A. 

Muddiman,  Matt  W. 

Seusy,  Cliff 
Spangler,  Lester 
Tracy,  Terry  A. 

Weber,  David  P. 


Director,  Center  of  Excellence 

for  R&M 

AFIT/CERM 

Former  F-16  Engine  Manager 
ASD/ENE  (GIMADS) 

OASD  (P&L/MD) 

Pentagon 


PhD  Fellow  Engineer 

Westinghouse  Electric  Corp. 

Supervisor 

Reliability  Methods  Group 
AT&T  Bell  Laboratories 

Manager:  Reliability  Engineering 

The  Analytic  Sciences  Corp. 

(TASC) 

District  Mgt-R&M  Methods 
Bellcore 

Staff  Engineer 
Product  Reliability 
Product/Process  Assurance 
Delco  Electronics 
-subsidiary  of  GM  Hughes 
Electronics 

Statistical  Development  Corp. 
Product  &  Process  Assurance 
Delco  Electronics  Corp. 

Quality  Engineer 
Hewlett  Packard 

Field  Engineer 

Dynamics  Research  Corp.  (DRC) 

Product/Process  Assurance 
Product  Reliability  Supervisor 
Delco  Electronics 

Lead  Engineer 

Reliability  &  Safety  Engineering 
Aircraft  Engine  Business  Group 
General  Electric  (GE) 
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Appendix  E: 


Average  7.17 
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Appendix  F :  First  Round  One  of  23  Interviewees  fDoDI 
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#  of  Resp. 


Average 
#  of  Resp. 
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8 

7 

5 

9 

9 

8 

6 

8 

9 

7 

6 

8 

9  . 

7 

5 

8 

8 

6 

8 

8 

6 

7 

7 

7 

7 

8.77 

9.07 

8.29 

7.83 

7.56 

9.8 

7.8 

17 

15 

14 

12 

9 

5 

5 

HUMAN 

POLITICS 

S/W 

ACQ.P. 

RQMT. 

8 

10 

6 

10 

9 

7 

8 

6 

7 

6 

7 

7 

6 

6 

4 

7.00 

8.33 

5.5 

8.5 

7.5 

4 

3 

4 

2 

2 
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Appendix  G: 


MFG  DESIGN  ENVMT  PREDS  HUMAN  TRNG  DATA 

10  10  9  10  8  10  9 

10  9877 

9  8  6 

Average  9.67  9.0  7.67  8.5  7.5  10  9 

#  of  Resp.  3  3  3  2  211 


MGMT 

8 

Average  8 

#  of  Resp.  1 
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Appendix  H:  First  Round  Two  of  28  Interviewees 
(Pop  &  Industry^ 


Average 
#  of  Resp 


PATA 

ENVMT 

PREPS 

MFG 

PESIGN 

MGMT 

TRNG 

10 

10 

10 

10 

10 
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10 

9 
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10 
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10 
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10 
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7 

9 

9 

8 

8 

7 

8 

9 

8 

7 

7 

8 

9 

8 

7 

6 

8 

8 

7 

7 

8 

8 

7 

6 

8 

8 

7 

6 

7 

7 

6 

7 

6 

6 

6 

5 

8.67 

8.56 

8.29 

8.33 

8.17 

9.57 

8.33 

18 

18 

17 

15 

12 

7 

6 

HUMAN  POLITICS  S/W  ACQ.P.  RQMT. 


8 

8 

8 

8 

7 

6 


10  6  10  9 

8  6  7  6 

7  6 

5 


Average  7 . 5 

#  of  Resp.  6 


8.33  5.75 
3  4 


8.5  7.5 

2  2 
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DATA 

ENVMT 

PREDS 

MFG 

DESIGN 

MGMT 

TRNG 

10 

10 

10 

10 

10 

10 

9 

10 

10 

10 

10 

9 

10 

8 

10 

10 

10 

9 

9 

10 

8 

10 

10 

9 

9 

8 

10 

8 

10 

9 

9 

9 

8 

10 

7 

10 

9 

9 

9 

7 

9 

9  9  9  8  7 

9  9  8  7  7 

9  9  8  7  6 

8  9  8  7 

8  9  8  6 

8  9  7  6 

8  8  7 

8  8  6 

7  5  6 

7 
6 

Average  8.65  8.87  8.27  8.08  7.89  9.83  8.00 

#  of  Resp.  17  15  15  12  9  65 


HUMAN  POLITICS  S/W  ACQ.P.  RQMT . 

8  10  6  10  9 

8  8  6  7  6 

?  7  6 

7  5 

Average  7.75  8.33  5.75  8.50  7.50 

#  of  Resp.  4  3422 
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Appendix  J :  First  Round  Two  of  5  Interviewees 


MFG  DESIGN  ENVMT  PREDS  HUMAN  TRNG  DATA 

10  9  9  1C  8  10  9 

10  9  7  7  6 

8  9  6 

Average  9.33  9.0  7.33  8.5  7.0  10  9 

#  of  Resp.  3  332211 


MGMT 

8 

Average  8 

#  of  Resp.  l 
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Appendix  K: 
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8 
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9  9  9  9  7  8 

9  9  9  9  7  8 

9  9  9  8  6  6 

9  9  9  8  5 

9  9  8  8 

9  9  8  7 

8  9  8  7 

8  9  8  7 

8  8  8  6 

8  8  7  6 

8  8  7  6 

8  7  7  5 

7  7  7 

7  7  6 

7  6  6 

7  6 

7 
5 

Average  8.56  8.79  8.36  8.19  8.15  8.67  8.17 

#  Of  Resp.  27  24  25  21  13  12  6 


HUMAN  POLITICS  S/W  P,H,&T  RQMT. 

8  10  6  99 

7  8  6  9  6 

7  7  6 

7  4 

6 

Average  7.0  8.33  5.5  9.0  7.5 

#  of  Resp.  5  34  22 
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Appendix  L:  Round  One  of  28  Interviewees  fDoD^ 
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7 

6 

8 

9 

7 

5 

8 

8 

6 

8 

8 

6 

8 

7 

6 

7 

7 

7 

7 

7 

5 

Average  8.46  9.11  8.33  7.80  7.90  9.00  7.80 

#  of  Resp.  24  18  18  15  10  8  5 


POLITICS  S/W  HUMAN  RQMT. 


Average 
#  of  Resp. 


10  6  7  9 

8  6  7  6 

7  6  6 

4 


8.33  5.5  6.67 

3  4  3 


7.5 

2 
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Appendix  M 


Round  one  of  10  Interviewees  (Industry! 


Average 
#  of  Resp 


Average 
#  of  Resp 


PREDS  MFG  ENVMT  MGMT  DATA  DESIGN  P,H,&T 


10  10  10 

10  10  9 

9  10  8 

8  10  7 

8  9  7 

7  6  6 

7 


8  10  10 

8  9  9 

8  9  8 

8 


9 

9 


8.43  9.17 
7  6 


7.83  8.00  9.33 
6  4  3 


9.00  9.00 

3  2 


HUMAN  TRNG  RQMT. 

8  10  9 

7 

7.5  10  9 

2  11 
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Average  8.48  8.35  8.54  8.29  8.79  8.29  8.33 

#  of  Resp.  27  26  24  21  14  14  6 


HUMAN  POLITICS  S/W  P,H,&T  RQMT. 

8  10  6  9  9 

8  8  6  9  6 
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7  5 

6 

Average  7.40  8.33  5.75  9.00  7.50 

#  of  Resp.  5  3422 
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Average  8.42  8.83  8.32  8.00  8.09  9.00  8.00 

#  of  Resp.  24  18  19  15  11  9  5 


POLITICS  S/W  HUMAN  RQMT. 
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#  of  Resp.  3432 


139 


Appendix  P: 


Round  Two  of  10  Interviewees  (industry l 
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Appendix  Q:  Selective  R&M  Data  Deficiencies  Tiger 


The  Fifteen  Problem  Descuriptions  are: 


1.  The  Air  Force  lacks  an  overall  focal  point  for 
logistics-related  data  systems.  Responsibility  and 
authority  is  fragmented  in  terms  of  requirements,  design 
responsibility,  and  resources. 

2.  R&M  terms  and  definitions  vary  in  A.ir  Force  documents 
(e.g.,  MIL-STD-721C  and  AFP  57-9;  and  in  the  Reliability  & 
Maintainability  Information  System  (REM13)  Functional 
Description  (FD) . 

3 .  In  order  to  measure  and  report  the  status  of  weapon 
system  reliability  and  maintainability,  and  to  assess  the 
impact  of  R&M  modifications  relative  to  R&M  2300  goaJs, 
there  must  be  a  standard  set  of  R&M  indicators  from  which 
analysis  can  be  performed. 

4.  There  is  no  structure  to  the  process  of  identifying  raw 
data  elements  which  are  necessary  for  identification  and 
analysis  of  R&M  problems  in  the  Air  Force. 

5.  Policies  in  the  Air  Force  do  not  adequately  cover  serial 
number  tracking  to  derive  reliability  information. 

6.  Present  field  data  systems  do  not  capture  the  necessary 
BIT  data  to  effectively  identify  fault-isolation  problems. 

7.  Currently  depot  maintenance  does  not  report  within  the 
Maintenance  Data  Collection  (MDC)  systems, 

8 .  The  current  MDC  system  and  the  Core  Automated 
Maintenance  system  (CAMS)  experience  «  loss  of  some  of  the 
on-equipment  data. 

9.  The  CAMS/REMIS  data  outputs  will  not  be  available  to  the 
acquisition  community  until  1992  or  later.  The  present  R&M 
data  systems  (AFTO  Form  349,  D056B,  D056T,  and  [Maintenance 
Operations  Data  Access  System]  MODAS) ,  could  be* 
significantly  improved  with  a  coordinated  effort,  thus 
"filling-the-gap"  until  REMIS  becomes  operational, 

10.  Several  weapon  systems  currently  have  the  capability  to 
provide  Maintenance  Integrated  Data  Access  System  (MIDAS) 
and  Functionally  Integrated  Designating  and  Referencing 
(FINDER)  information.  However,  our  current  maintenance  data 
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collection  system  is  not  geared  towards  collecting 
maintenance  fault  codes,  job  guide  function  numbers,  or 
reference  designators. 

11.  The  Air  Force  R&K  community,  as  well  as  contractors,  do 
not  have  access  to  the  data  that  will  provide  the  effect  of 
failures  on  weapon  system  capability  (i.e.,  critical 
failure) . 

12.  Data  in  the  current  MDC  systems  are  suspect. 

13.  CAMS  has  a  limited  MDC  correction  capability.  Early 
experience  with  CAME  at  Dyess  and  Ellsworth  AFBs  shows  a 
high  MDC.  error  rate  (in  the  3  0  percent  range)  .  There  is  a 
need  to  be  able  to  correct  these  data  prior  to  their 
transmission  to  AFLC. 


14.  While  there  arc  a  vast  amount  of  R&M  data,  accurate 
failure/fix  information  is  not  collected  due  to  the  failure 
of  the  organisational  and  intermediate  level  personnel  to 
properly  document  their  actions.  There  is  a  significant 
amount  of  emphasis  across  the  operational  commands  on  time 
accounting.  This  emphasis  does  not  originate  from  the 
KKTCOMS.  However,  no  amount  of  contrary  information  has 
been  able  to  change  the  perception  that  it  is  necessary  to 
document  "8  hrs/day  per  man.” 

15.  The  present  data  systems  cannot  capture  R&M  Time  Stress 
Measurement  Device  (TSMD)  data  that  will  be  available  on  all 
LRU  *  s  by  1990.  The  data  could  be  useful  for  warranty  and 
R&M  administration. 
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Appendix  R:  Relative  Relationship  Between  Reliabilit-\ 
Parameters  166:1061 
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1.  To  examine  the  existence  of  the  reliability  gap 
in  airborne  tactical  radars. 

2.  To  identify  and  analyze  the  major  contributors 

to  the  reliability  gap.  Specif ically,  to  identify 
the  most  significant  contributors. 

3.  To  explore  practical  guidelines  on  how  to  minimize 
the  reliability  gap.  Specifically,  to  identify 
some  practical  guidelines  on  reliability  growth. 

To  examine  the  existence  of  the  reliability  gap,  this 
thesis  obtained  the  data  from  NAVA I R  and  examined  the 
radars  on  Air  Force  and  Navy  aircraft.  Using  experts'/ 
practitioners'  (DoD  &  Industry)  opinions,  contributors  to 
the  reliability  gap  were  identified.  In  addition,  some 
practical  guidelines  on  reliability  growth  management  were 
also  identified,^  This  thesis  consolidated  some  of  the 
best  currently  available  thinking  on  the  major  contributors 
to  the  reliability  gap  and  some  of  the  practical  guidelines 
on  reliability  grpwth. 
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